Attachment 853763 Details for Bug 1056480 – the thread-dump logs in development.log

the thread-dump logs in development.log

develop_threaddump.log (text/x-log), 16.39 KB, created by chunchen on 2014-01-22 09:42:37 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: chunchen

Created: 2014-01-22 09:42:37 UTC

Size: 16.39 KB

patch

obsolete

>2014-01-22 04:23:37.485 [INFO ] Started GET "/broker/rest/api" for 203.114.244.88 at 2014-01-22 04:23:37 -0500 (pid:2337)
>2014-01-22 04:23:37.487 [INFO ] Processing by ApiController#show as JSON (pid:2337)
>2014-01-22 04:23:37.488 [DEBUG] API version 1.3 (pid:2337)
>2014-01-22 04:23:37.498 [INFO ] Completed 200 OK in 11ms (Views: 5.8ms) (pid:2337)
>2014-01-22 04:23:37.861 [INFO ] Started GET "/broker/rest/api" for 203.114.244.88 at 2014-01-22 04:23:37 -0500 (pid:2337)
>2014-01-22 04:23:37.863 [INFO ] Processing by ApiController#show as JSON (pid:2337)
>2014-01-22 04:23:37.863 [DEBUG] API version 1.6 (pid:2337)
>2014-01-22 04:23:37.875 [INFO ] Completed 200 OK in 12ms (Views: 6.5ms) (pid:2337)
>2014-01-22 04:23:38.230 [INFO ] Started GET "/broker/rest/domains" for 203.114.244.88 at 2014-01-22 04:23:38 -0500 (pid:2337)
>2014-01-22 04:23:38.232 [INFO ] Processing by DomainsController#index as JSON (pid:2337)
>2014-01-22 04:23:38.233 [DEBUG] API version 1.6 (pid:2337)
>2014-01-22 04:23:38.237 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=52df72171464691832000001 LOGIN=chunchen IP=203.114.244.88 SCOPES=session Authenticated (pid:2337)
>2014-01-22 04:23:38.243 [DEBUG] SUCCESS ACTION=LIST_DOMAIN USER_ID=52df72171464691832000001 LOGIN=chunchen DOMAIN= (pid:2337)
>2014-01-22 04:23:38.250 [INFO ] Completed 200 OK in 18ms (Views: 5.8ms) (pid:2337)
>2014-01-22 04:23:38.610 [INFO ] Started GET "/broker/rest/domains/cdm/applications/crb19" for 203.114.244.88 at 2014-01-22 04:23:38 -0500 (pid:2337)
>2014-01-22 04:23:38.613 [INFO ] Processing by ApplicationsController#show as JSON (pid:2337)
>2014-01-22 04:23:38.613 [INFO ] Parameters: {"domain_id"=>"cdm", "id"=>"crb19"} (pid:2337)
>2014-01-22 04:23:38.613 [DEBUG] API version 1.6 (pid:2337)
>2014-01-22 04:23:38.617 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=52df72171464691832000001 LOGIN=chunchen IP=203.114.244.88 SCOPES=session Authenticated (pid:2337)
>2014-01-22 04:23:38.703 [DEBUG] SUCCESS ACTION=SHOW_APPLICATION USER_ID=52df72171464691832000001 LOGIN=chunchen APP_UUID=52df8ce114646918320000a9 DOMAIN=cdm Application 'crb19' found (pid:2337)
>2014-01-22 04:23:38.721 [INFO ] Completed 200 OK in 108ms (Views: 16.1ms) (pid:2337)
>2014-01-22 04:23:39.423 [INFO ] Started POST "/broker/rest/application/52df8ce114646918320000a9/events" for 203.114.244.88 at 2014-01-22 04:23:39 -0500 (pid:2337)
>2014-01-22 04:23:39.427 [INFO ] Processing by AppEventsController#create as JSON (pid:2337)
>2014-01-22 04:23:39.427 [INFO ] Parameters: {"event"=>"thread-dump", "application_id"=>"52df8ce114646918320000a9", "app_event"=>{"event"=>"thread-dump"}} (pid:2337)
>2014-01-22 04:23:39.428 [DEBUG] API version 1.6 (pid:2337)
>2014-01-22 04:23:39.432 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=52df72171464691832000001 LOGIN=chunchen IP=203.114.244.88 SCOPES=session Authenticated (pid:2337)
>2014-01-22 04:23:39.441 [DEBUG] DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"ruby-1.9", :action=>"threaddump", :args=>{"--with-app-uuid"=>"52df8ce114646918320000a9", "--with-app-name"=>"crb19", "--with-container-uuid"=>"52df8ce114646918320000a9", "--with-container-name"=>"crb19", "--with-namespace"=>"cdm", "--with-request-id"=>"3d6c9ac39743fb99fc60ba4219c4ba57", "--cart-name"=>"ruby-1.9", "--component-name"=>"ruby-1.9", "--with-software-version"=>"1.9", "--cartridge-vendor"=>"redhat"}}, ip-10-180-231-119, {'identity' => ip-10-180-231-119}) (Request ID: 3d6c9ac39743fb99fc60ba4219c4ba57) (pid:2337)
>2014-01-22 04:23:39.868 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x0000000798d9b8 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-180-231-119", :statuscode=>5, :statusmsg=>"undefined local variable or method `i' for #<MCollective::Agent::Openshift:0x000000022b8688>", :data=>{:time=>nil, :output=>nil, :exitcode=>nil, :addtl_params=>nil}}>] (Request ID: 3d6c9ac39743fb99fc60ba4219c4ba57) (pid:2337)
>2014-01-22 04:23:39.871 [DEBUG] DEBUG: MCollective Response Time (execute_direct: threaddump): 0.431515865s  (Request ID: 3d6c9ac39743fb99fc60ba4219c4ba57) (pid:2337)
>2014-01-22 04:23:39.871 [ERROR] An invalid exit code (nil) was returned from the server ip-10-180-231-119.  This indicates an unexpected problem during the execution of your request. (pid:2337)
>2014-01-22 04:23:39.871 [ERROR] #<OpenShift::NodeException: An invalid exit code (nil) was returned from the server ip-10-180-231-119.  This indicates an unexpected problem during the execution of your request.> (pid:2337)
>2014-01-22 04:23:39.872 [ERROR] ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.19.10/lib/openshift/mcollective_application_container_proxy.rb:2733:in `parse_result'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.19.10/lib/openshift/mcollective_application_container_proxy.rb:2901:in `run_cartridge_command'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.19.10/lib/openshift/mcollective_application_container_proxy.rb:1148:in `threaddump'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/gear.rb:282:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/application.rb:1014:in `block (2 levels) in threaddump'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/group_instance.rb:197:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/group_instance.rb:197:in `block in run_on_gears'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/group_instance.rb:195:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/group_instance.rb:195:in `run_on_gears'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/application.rb:1013:in `block in threaddump'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/models/application.rb:1010:in `threaddump'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/app/controllers/app_events_controller.rb:93:in `create'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:492:in `block in _run__3268646558851064708__process_action__574089132249402744__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_486'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.19.13/lib/openshift/controller/action_log.rb:80:in `set_logged_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_486'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__3268646558851064708__process_action__574089132249402744__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'", "/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__2529903892857910090__call__2243878344898116501__callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'", "/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'", "/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>'"] (pid:2337)
>2014-01-22 04:23:39.889 [DEBUG] SUCCESS ACTION=THREAD_DUMP_APPLICATION USER_ID=52df72171464691832000001 LOGIN=chunchen APP_UUID=52df8ce114646918320000a9 DOMAIN=cdm (pid:2337)
>2014-01-22 04:23:39.902 [INFO ] Completed 200 OK in 475ms (Views: 12.2ms) (pid:2337)

Actions: View

Attachments on bug 1056480: 853763