Description of problem: I imported oct's manifest from cli and started the sync for rhel6 rpm and rhel6 option rpm repos. When I click on sync status to check the sync status, UI throws following error: -- 500 Internal Server Error Sorry about that! It seems something went wrong. If you continue having trouble with this, please contact an Administrator. You might want to log out and log back in again to clear your cookies. Error: Pulp::Repository: Request Timeout (GET /pulp/api/repositories/?_intersect=groupid&groupid=product:69&groupid=env:34) --- Also when I checked the grinder logs under /var/log/pulp/grinder.log, the sync was in-progress state. Version-Release number of selected component (if applicable): [root@dhcp201-198 pulp]# rpm -qa | grep katello katello-common-0.1.98-1.git.33.a663155.fc15.noarch katello-all-0.1.98-1.git.33.a663155.fc15.x86_64 katello-cli-common-0.1.10-1.git.1133.ea1fcc6.fc15.noarch katello-glue-foreman-0.1.98-1.git.33.a663155.fc15.x86_64 katello-configure-0.1.10-1.git.0.78fbe70.fc15.noarch katello-cli-0.1.10-1.git.1133.ea1fcc6.fc15.noarch katello-glue-pulp-0.1.98-1.git.33.a663155.fc15.x86_64 katello-0.1.99-1.git.89.bc81fa8.fc15.x86_64 katello-glue-candlepin-0.1.98-1.git.33.a663155.fc15.x86_64 katello-repos-0.1.3-1.git.0.db2bd1d.fc15.noarch [root@dhcp201-198 pulp]# rpm -qa | grep pulp mod_wsgi-3.2-6.pulp.fc15.x86_64 pulp-0.0.244-5.fc15.noarch m2crypto-0.21.1.pulp-5.fc15.x86_64 katello-glue-pulp-0.1.98-1.git.33.a663155.fc15.x86_64 pulp-common-0.0.244-5.fc15.noarch [root@dhcp201-198 pulp]# How reproducible: Steps to Reproduce: 1. import oct's manifest 2. start sync for 2 repos 3. Check the sync status after some time. Actual results: Pulp::Repository: Request Timeout (GET /pulp/api/repositories/?_intersect=groupid&groupid=product:69&groupid=env:34) Expected results: UI should show the sync status without any error. Additional info: production.log says: ===================== Started GET "/katello//sync_management/index" for 10.65.193.48 at Wed Nov 09 14:55:52 +0530 2011 Processing by SyncManagementController#index as HTML Pulp::Repository: Request Timeout (GET /pulp/api/repositories/?_intersect=groupid&groupid=product:69&groupid=env:34) Pulp::Repository: Request Timeout (GET /pulp/api/repositories/?_intersect=groupid&groupid=product:69&groupid=env:34) Pulp::Repository: Request Timeout (GET /pulp/api/repositories/?_intersect=groupid&groupid=product:69&groupid=env:34): /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:174:in `transmit' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:54:in `get' /usr/share/katello/lib/http_resource.rb:92:in `get' /usr/share/katello/lib/resources/pulp.rb:181:in `all' /usr/share/katello/app/models/glue/pulp/repos.rb:473:in `cache_repos' /usr/share/katello/app/models/glue/pulp/repos.rb:482:in `get_cached_repos' /usr/share/katello/app/models/glue/pulp/repos.rb:79:in `repos' /usr/share/katello/app/controllers/sync_management_controller.rb:59:in `index' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:366:in `reject' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:366:in `send' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/relation.rb:366:in `method_missing' /usr/share/katello/app/controllers/sync_management_controller.rb:59:in `index' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:150:in `process_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:11:in `process_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:18:in `process_action' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:457:in `_run__1945219280__process_action__199225275__callbacks' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:221:in `_conditional_callback_around_1977' /usr/share/katello/lib/util/threadsession.rb:79:in `thread_locals' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:220:in `_conditional_callback_around_1977' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:451:in `_run__1945219280__process_action__199225275__callbacks' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `send' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `_run_process_action_callbacks' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `send' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `run_callbacks' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:17:in `process_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:17:in `process_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30:in `process_action' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21:in `instrument' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:29:in `process_action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:119:in `process' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:41:in `process' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:138:in `dispatch' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:178:in `action' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `dispatch' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:27:in `call' /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:150:in `call' /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:93:in `recognize' /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:68:in `optimized_each' /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:92:in `recognize' /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:141:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:493:in `call' /usr/lib/ruby/gems/1.8/gems/warden-1.0.3/lib/warden/manager.rb:35:in `call' /usr/lib/ruby/gems/1.8/gems/warden-1.0.3/lib/warden/manager.rb:34:in `catch' /usr/lib/ruby/gems/1.8/gems/warden-1.0.3/lib/warden/manager.rb:34:in `call' /usr/lib/ruby/gems/1.8/gems/sass-3.1.10/lib/sass/../sass/plugin/rack.rb:54:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/head.rb:14:in `call' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/params_parser.rb:21:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:182:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:149:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302:in `call' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:32:in `call' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:28:in `cache' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:12:in `cache' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:31:in `call' /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:354:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:46:in `call' /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416:in `_run_call_callbacks' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:44:in `call' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/remote_ip.rb:48:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47:in `call' /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:13:in `call' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17:in `call' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:34:in `call' /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/static.rb:30:in `call' /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:168:in `call' /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `send' /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `method_missing' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:52:in `call' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46:in `each' /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46:in `call' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57:in `process' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!' /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/bin/thin:6 /usr/share/katello/script/thin:59:in `load' /usr/share/katello/script/thin:59 Rendering 500:Pulp::Repository: Request Timeout (GET /pulp/api/repositories/?_intersect=groupid&groupid=product:69&groupid=env:34) Rendered common/_config.html.haml (0.4ms) Rendered layouts/_ajax_notices.haml (2.5ms) Rendered layouts/_notification.haml (0.3ms) Rendered common/_common_i18n.html.haml (0.2ms) Rendered layouts/_org.haml (0.8ms) Rendered layouts/_header.haml (5.2ms) Rendered layouts/_footer.haml (88.8ms) Rendered common/500.html.haml within layouts/katello (476.5ms) Completed 500 Internal Server Error in 30826ms (Views: 459.9ms | ActiveRecord: 33.7ms)
I can confirm I saw this as well after syncing 3 base RHEL channels at the same time.
setup multiple machines with the latest code and synced between 3-10 repos at the same time. On slow hardware the syncs took a while but I never saw any timeout issues On fast hardware the syncs were super fast (7GB in like 15min) and no timeout errors. Flipping this to MODIFIED.
mass ON_QA move
Marking this verified. I haven't been able to reproduce any timeouts lately when viewing sync status. Can reopen if they begin recurring.
http://www.redhat.com/archives/spacewalk-list/2011-September/msg00191.html has some useful generic PostgreSQL tuning hints which may help with 500 Internal Server errors.