Bug 988699

Summary: Can't restart a CapeDwarf application from website
Product: OpenShift Online Reporter: Tian Feng <tfeng>
Component: ContainersAssignee: Dan Mace <dmace>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.xCC: dmcphers, jhonce, mlazar, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-07 22:56:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tian Feng 2013-07-26 07:31:54 UTC 
Description of problem:

When I restart a CapeDwarf application from website, it would return error, and the application can't be accessed any more.


Version-Release number of selected component (if applicable):
INT(devenv_3561)

How reproducible:
Always

Steps to Reproduce:
1. Create a CapeDwarf application and check the application page
2. Restart it from website
3.

Actual results:

After restart it, it would return  "We appear to be having technical difficulties" error page, and the application page returns 503

Expected results:

The CapeDwarf application should be restarted successfully.

Additional info:
Comment 1 Tian Feng 2013-07-26 07:33:59 UTC 
Ssh into the devenv and trace the log, here is the error message

2013-07-26 03:26:08.733 [INFO ] Started GET "/broker/rest/domain/skldfj/application/capedwarf.json" for 127.0.0.1 at 2013-07-26 03:26:08 -0400 (pid:4032)
2013-07-26 03:26:08.735 [INFO ] Processing by ApplicationsController#show as JSON (pid:4032)
2013-07-26 03:26:08.735 [INFO ] Parameters: {"domain_id"=>"skldfj", "id"=>"capedwarf"} (pid:4032)
2013-07-26 03:26:08.737 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=authorizations selector={"$query"=>{"token"=>"70cf895a1228a2f9a5b5f66d416735f041acbd8658f1eef8c1c7b3facd2da701"}, "$orderby"=>{:_id=>1}} flags=[:slave_ok] limit=-1 skip=0 batch_size=nil fields=nil (0.8154ms) (pid:4032)
2013-07-26 03:26:08.739 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=cloud_users selector={"$query"=>{"_id"=>"51f2106b3b34516264000001"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.5953ms) (pid:4032)
2013-07-26 03:26:08.740 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=51f2106b3b34516264000001 LOGIN=tfeng+july22 IP=127.0.0.1 SCOPES=session Authenticated (pid:4032)
2013-07-26 03:26:08.742 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=domains selector={"$query"=>{"owner_id"=>"51f2106b3b34516264000001", "canonical_namespace"=>"skldfj"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.9220ms) (pid:4032)
2013-07-26 03:26:08.745 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=applications selector={"$query"=>{"domain_id"=>"51f210993b34516264000004", "canonical_name"=>"capedwarf"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (1.1277ms) (pid:4032)
2013-07-26 03:26:08.749 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=domains selector={"$query"=>{"_id"=>"51f210993b34516264000004"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.3960ms) (pid:4032)
2013-07-26 03:26:08.751 [DEBUG] SUCCESS ACTION=SHOW_APPLICATION USER_ID=51f2106b3b34516264000001 LOGIN=tfeng+july22 APP=capedwarf DOMAIN=skldfj APP_UUID=36846a28f5c411e2a58c22000a988338 Application 'capedwarf' found (pid:4032)
2013-07-26 03:26:08.753 [INFO ] Completed 200 OK in 18ms (Views: 1.0ms) (pid:4032)
2013-07-26 03:26:08.801 [INFO ] Started POST "/broker/rest/domain/skldfj/application/capedwarf/events.json" for 127.0.0.1 at 2013-07-26 03:26:08 -0400 (pid:4032)
2013-07-26 03:26:08.804 [INFO ] Processing by AppEventsController#create as JSON (pid:4032)
2013-07-26 03:26:08.805 [INFO ] Parameters: {"event"=>"restart", "domain_id"=>"skldfj", "application_id"=>"capedwarf", "app_event"=>{"event"=>"restart"}} (pid:4032)
2013-07-26 03:26:08.806 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=authorizations selector={"$query"=>{"token"=>"70cf895a1228a2f9a5b5f66d416735f041acbd8658f1eef8c1c7b3facd2da701"}, "$orderby"=>{:_id=>1}} flags=[:slave_ok] limit=-1 skip=0 batch_size=nil fields=nil (0.7663ms) (pid:4032)
2013-07-26 03:26:08.808 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=cloud_users selector={"$query"=>{"_id"=>"51f2106b3b34516264000001"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.5045ms) (pid:4032)
2013-07-26 03:26:08.809 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=51f2106b3b34516264000001 LOGIN=tfeng+july22 IP=127.0.0.1 SCOPES=session Authenticated (pid:4032)
2013-07-26 03:26:08.811 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=domains selector={"$query"=>{"owner_id"=>"51f2106b3b34516264000001", "canonical_namespace"=>"skldfj"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.7725ms) (pid:4032)
2013-07-26 03:26:08.813 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=applications selector={"$query"=>{"domain_id"=>"51f210993b34516264000004", "canonical_name"=>"capedwarf"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (1.0395ms) (pid:4032)
2013-07-26 03:26:08.815 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=domains selector={"$query"=>{"_id"=>"51f210993b34516264000004"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.4210ms) (pid:4032)
2013-07-26 03:26:08.818 [DEBUG] MOPED: 127.0.0.1:27017 COMMAND      database=openshift_broker_dev command={:findAndModify=>"locks", :query=>{"user_id"=>"51f2106b3b34516264000001", "$or"=>[{"app_ids.51f223d93b34514bd5000011"=>{"$exists"=>false}}, {"app_ids.51f223d93b34514bd5000011"=>{"$lt"=>1374823568}}]}, :new=>true, :update=>{"$set"=>{"app_ids.51f223d93b34514bd5000011"=>1374825368}}} (0.9036ms) (pid:4032)
2013-07-26 03:26:08.821 [DEBUG] MOPED: 127.0.0.1:27017 UPDATE       database=openshift_broker_dev collection=applications selector={"_id"=>"51f223d93b34514bd5000011"} update={"$push"=>{"pending_op_groups"=>{"_id"=>"51f224903b34518e260000b8", "num_gears_added"=>0, "num_gears_removed"=>0, "num_gears_created"=>0, "num_gears_destroyed"=>0, "num_gears_rolled_back"=>0, "user_agent"=>nil, "op_type"=>:restart_app, "parent_op_id"=>nil, "updated_at"=>2013-07-26 07:26:08 UTC, "created_at"=>2013-07-26 07:26:08 UTC}}} flags=[] (pid:4032)
2013-07-26 03:26:08.821 [DEBUG] COMMAND      database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (0.8905ms) (pid:4032)
2013-07-26 03:26:08.822 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=applications selector={:_id=>"51f223d93b34514bd5000011"} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.5670ms) (pid:4032)
2013-07-26 03:26:08.827 [DEBUG] MOPED: 127.0.0.1:27017 UPDATE       database=openshift_broker_dev collection=applications selector={"_id"=>"51f223d93b34514bd5000011"} update={"$push"=>{"pending_op_groups.0.pending_ops"=>{"_id"=>"51f224903b34518e260000b9", "state"=>:init, "retry_count"=>0, "saved_values"=>[], "op_type"=>:restart_component, "args"=>{"group_instance_id"=>"51f223d93b34514bd500001c", "gear_id"=>"51f223d93b34514bd5000011", "comp_spec"=>{"cart"=>"jbossas-7", "comp"=>"jbossas-7"}}}}} flags=[] (pid:4032)
2013-07-26 03:26:08.828 [DEBUG] COMMAND      database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (0.7441ms) (pid:4032)
2013-07-26 03:26:08.829 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=applications selector={"$query"=>{"_id"=>"51f223d93b34514bd5000011"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.6406ms) (pid:4032)
2013-07-26 03:26:08.832 [DEBUG] MOPED: 127.0.0.1:27017 UPDATE       database=openshift_broker_dev collection=applications selector={"_id"=>"51f223d93b34514bd5000011"} update={"$set"=>{"pending_op_groups.0.pending_ops"=>[{"_id"=>"51f224903b34518e260000b9", "state"=>:init, "retry_count"=>0, "saved_values"=>[], "op_type"=>:restart_component, "args"=>{"group_instance_id"=>"51f223d93b34514bd500001c", "gear_id"=>"51f223d93b34514bd5000011", "comp_spec"=>{"cart"=>"jbossas-7", "comp"=>"jbossas-7"}}}]}} flags=[] (pid:4032)
2013-07-26 03:26:08.832 [DEBUG] COMMAND      database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (0.6249ms) (pid:4032)
2013-07-26 03:26:08.834 [DEBUG] Execute restart_component (pid:4032)
2013-07-26 03:26:08.836 [DEBUG] MOPED: 127.0.0.1:27017 UPDATE       database=openshift_broker_dev collection=applications selector={"_id"=>"51f223d93b34514bd5000011"} update={"$set"=>{"pending_op_groups.0.pending_ops.0.state"=>:queued}} flags=[] (pid:4032)
2013-07-26 03:26:08.836 [DEBUG] COMMAND      database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (0.7579ms) (pid:4032)
2013-07-26 03:26:08.837 [DEBUG] MOPED: 127.0.0.1:27017 QUERY        database=openshift_broker_dev collection=domains selector={"$query"=>{"_id"=>"51f210993b34516264000004"}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil (0.4008ms) (pid:4032)
2013-07-26 03:27:48.985 [DEBUG] DEBUG: Output of parallel execute: [{:tag=>{"op_id"=>"51f224903b34518e260000b9"}, :gear=>"36846a28f5c411e2a58c22000a988338", :job=>{:cartridge=>"openshift-origin-node", :action=>"restart", :args=>{"--with-app-uuid"=>"36846a28f5c411e2a58c22000a988338", "--with-app-name"=>"capedwarf", "--with-container-uuid"=>"36846a28f5c411e2a58c22000a988338", "--with-container-name"=>"capedwarf", "--with-namespace"=>"skldfj", "--with-request-id"=>"97e3d524fcdbf7325e840656b8f53467", "--cart-name"=>"jbossas-7", "--component-name"=>"jbossas-7", "--with-software-version"=>"7", "--cartridge-vendor"=>"redhat"}}, :result_stdout=>"Failed to execute: 'control restart' for /var/lib/openshift/36846a28f5c411e2a58c22000a988338/jbossas\nRestarting jbossas cart\nStopping jbossas cart\nStarting jbossas cart\nTimed out waiting for http listening port\n\nSending SIGTERM to jboss:6124 ...\n", :result_stderr=>"", :result_exit_code=>1}], exitcode: 0, from: ip-10-152-131-56  (Request ID: 97e3d524fcdbf7325e840656b8f53467) (pid:4032)
2013-07-26 03:27:48.990 [DEBUG] DEBUG: MCollective Response Time (execute_parallel): 100.152415922s  (Request ID: 97e3d524fcdbf7325e840656b8f53467) (pid:4032)
2013-07-26 03:27:48.991 [DEBUG] DEBUG: server results: Failed to execute: 'control restart' for /var/lib/openshift/36846a28f5c411e2a58c22000a988338/jbossas (pid:4032)
2013-07-26 03:27:48.991 [DEBUG] DEBUG: server results: Restarting jbossas cart (pid:4032)
2013-07-26 03:27:48.991 [DEBUG] DEBUG: server results: Stopping jbossas cart (pid:4032)
2013-07-26 03:27:48.991 [DEBUG] DEBUG: server results: Starting jbossas cart (pid:4032)
2013-07-26 03:27:48.991 [DEBUG] DEBUG: server results: Timed out waiting for http listening port (pid:4032)
2013-07-26 03:27:48.992 [DEBUG] DEBUG: server results: (pid:4032)
2013-07-26 03:27:48.992 [DEBUG] DEBUG: server results: Sending SIGTERM to jboss:6124 ... (pid:4032)
2013-07-26 03:27:48.995 [DEBUG] MOPED: 127.0.0.1:27017 UPDATE       database=openshift_broker_dev collection=applications selector={"_id"=>"51f223d93b34514bd5000011"} update={"$set"=>{"pending_op_groups.0.pending_ops.0.state"=>:failed}} flags=[] (pid:4032)
2013-07-26 03:27:48.995 [DEBUG] COMMAND      database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (0.8221ms) (pid:4032)
2013-07-26 03:27:49.536 [ERROR] Failed to correctly execute all parallel operations (pid:4032)
2013-07-26 03:27:49.598 [ERROR] ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/pending_app_op_group.rb:372:in `execute'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1267:in `run_jobs'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:779:in `block in restart'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1309:in `run_in_application_lock'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:770:in `restart'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/controllers/app_events_controller.rb:56:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:470:in `block in _run__4193444968238431067__process_action__1280270883853522562__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_340'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/lib/openshift/controller/action_log.rb:80:in `set_logged_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_340'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__4193444968238431067__process_action__1280270883853522562__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__652126739049666308__call__1522501945485491899__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>'"] (pid:4032)
2013-07-26 03:27:49.622 [ERROR] Failed to correctly execute all parallel operations (pid:4032)
2013-07-26 03:27:49.663 [DEBUG] ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/pending_app_op_group.rb:372:in `execute'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1267:in `run_jobs'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:779:in `block in restart'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1309:in `run_in_application_lock'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:770:in `restart'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/controllers/app_events_controller.rb:56:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:470:in `block in _run__4193444968238431067__process_action__1280270883853522562__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_340'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/lib/openshift/controller/action_log.rb:80:in `set_logged_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_340'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__4193444968238431067__process_action__1280270883853522562__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__652126739049666308__call__1522501945485491899__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>'"] (pid:4032)
2013-07-26 03:27:49.888 [DEBUG] MOPED: 127.0.0.1:27017 UPDATE       database=openshift_broker_dev collection=applications selector={"_id"=>"51f223d93b34514bd5000011"} update={"$pull"=>{"pending_op_groups"=>{"_id"=>"51f224903b34518e260000b8"}}} flags=[] (pid:4032)
2013-07-26 03:27:49.957 [DEBUG] COMMAND      database=openshift_broker_dev command={:getlasterror=>1, :safe=>true} (0.6530ms) (pid:4032)
2013-07-26 03:27:49.120 [DEBUG] MOPED: 127.0.0.1:27017 COMMAND      database=openshift_broker_dev command={:findAndModify=>"locks", :query=>{"user_id"=>"51f2106b3b34516264000001", "locked"=>false, "app_ids.51f223d93b34514bd5000011"=>{"$exists"=>true}}, :new=>true, :update=>{"$unset"=>{"app_ids.51f223d93b34514bd5000011"=>""}}} (1.3876ms) (pid:4032)
2013-07-26 03:27:49.135 [ERROR] Failed to correctly execute all parallel operations:["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/pending_app_op_group.rb:372:in `execute'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1267:in `run_jobs'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:779:in `block in restart'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1309:in `run_in_application_lock'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:770:in `restart'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/controllers/app_events_controller.rb:56:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:470:in `block in _run__4193444968238431067__process_action__1280270883853522562__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_340'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/lib/openshift/controller/action_log.rb:80:in `set_logged_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_340'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__4193444968238431067__process_action__1280270883853522562__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__652126739049666308__call__1522501945485491899__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>'"] (pid:4032)
2013-07-26 03:27:49.168 [ERROR] Reference ID: 97e3d524fcdbf7325e840656b8f53467 - Failed to correctly execute all parallel operations
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/pending_app_op_group.rb:372:in `execute'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1267:in `run_jobs'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:779:in `block in restart'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:1309:in `run_in_application_lock'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/models/application.rb:770:in `restart'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/app/controllers/app_events_controller.rb:56:in `create'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:470:in `block in _run__4193444968238431067__process_action__1280270883853522562__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_340'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.2/lib/openshift/controller/action_log.rb:80:in `set_logged_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_340'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__4193444968238431067__process_action__1280270883853522562__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__652126739049666308__call__1522501945485491899__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>' (pid:4032)
2013-07-26 03:27:49.176 [DEBUG] FAILURE ACTION=RESTART_APPLICATION USER_ID=51f2106b3b34516264000001 LOGIN=tfeng+july22 APP=capedwarf DOMAIN=skldfj APP_UUID=36846a28f5c411e2a58c22000a988338 Unable to complete the requested operation due to: Failed to correctly execute all parallel operations.
Reference ID: 97e3d524fcdbf7325e840656b8f53467 (pid:4032)
2013-07-26 03:27:49.200 [INFO ] Completed 500 Internal Server Error in 100215ms (Views: 0.6ms) (pid:4032)
Comment 2 Matej Lazar 2013-07-26 09:47:34 UTC 
Looks like .openshift/action_hooks/pre_start_jbossas is not called.
The same problem occur if you use rhc app restart.

Doing a git push to app repo, application is started successfully.
Comment 3 Dan Mace 2013-07-31 15:42:05 UTC 
The following patch seems to fix the quickstart by eliminating the use of /etc/openshift/resource_limits.conf:

https://github.com/ironcladlou/capedwarf-example/commit/39347ecc34be2b7fb414f832d49fbb9b911a2b9d

The PermGen scaling logic isn't quite the same as the previous code, but it might still be okay. Matej, do you see any problems with the way I'm computing the PermGen value? If not, this might be all we need.
Comment 4 Matej Lazar 2013-08-01 13:55:25 UTC 
Calculation seems ok, but restart is still not working.
Tested via "rhc app restart".

While it works with
"rhc app stop"
"rhc app start"

and with git push too.

According to errors pre_start_jbossas in not executed on restart.
Comment 5 Matej Lazar 2013-08-01 15:56:44 UTC 
fixed with restart hook
https://github.com/openshift/capedwarf-example/commit/46d8db3b3ee592995fc24d823a25478501b3f0ee
Comment 6 Tian Feng 2013-08-02 07:26:33 UTC 
Test on STG, it can be restarted successfully, so verified it.