Description of problem: While attempting a content sync, the katello CLI eventually fails ... > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0% > Progress: [ ] 0.0%<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 246) /var/log/katello/client.log shows the follow error in more detail: > 2012-05-16 19:32:29,999 [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> Version-Release number of selected component (if applicable): * candlepin-0.5.26-1.el6.src.rpm * katello-0.1.311-1.el6_2.src.rpm * katello-candlepin-cert-key-pair-1.0-1.src.rpm * katello-certs-tools-1.0.4-1.el6.src.rpm * katello-cli-0.1.107-1.el6.src.rpm * katello-configure-0.1.107-1.el6.src.rpm * katello-qpid-broker-key-pair-1.0-1.src.rpm * katello-qpid-client-key-pair-1.0-1.src.rpm * katello-selinux-0.1.10-1.el6.src.rpm * pulp-1.0.4-1.el6.src.rpm * redhat-release-server-6Server-6.2.0.3.el6.src.rpm * subscription-manager-0.96.18-1.el6_2.src.rpm How reproducible: * Seems to happen on my CFSE beaker tests when katello is installed from RHN + Does *NOT* happen using the same MANIFEST, when testing CFSE and installing katello from RHSM (same package) Steps to Reproduce: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: Create custom Org to work with :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: Running 'katello -u admin -p admin org create --name redhat' :: [ PASS ] :: Running 'katello -u admin -p admin client remember --option org --value redhat' :: [ LOG ] :: Duration: 2s :: [ LOG ] :: Assertions: 2 good, 0 bad :: [ PASS ] :: RESULT: Create custom Org to work with :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: Create custom Environments :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: Running 'katello -u admin -p admin environment create --name Dev --prior Library' :: [ PASS ] :: Running 'katello -u admin -p admin environment create --name Test --prior Dev' :: [ PASS ] :: Running 'katello -u admin -p admin environment create --name Prod --prior Test' :: [ PASS ] :: Running 'katello -u admin -p admin client remember --option environment --value Dev' :: [ PASS ] :: Running 'katello -u admin -p admin client remember --option env --value Dev' :: [ LOG ] :: Duration: 5s :: [ LOG ] :: Assertions: 5 good, 0 bad :: [ PASS ] :: RESULT: Create custom Environments :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: Import manifest :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: Running 'curl -o /tmp/manifest.zip http://file.rdu.redhat.com/jlaska/manifests_access.zip' :: [ PASS ] :: Running 'katello -u admin -p admin provider update --name "Red Hat" --url https://cdn.redhat.com' :: [ PASS ] :: Running 'katello -u admin -p admin provider import_manifest --name "Red Hat" --file /tmp/manifest.zip' :: [ LOG ] :: Duration: 20m 21s :: [ LOG ] :: Assertions: 3 good, 0 bad :: [ PASS ] :: RESULT: Import manifest :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: Enabling 'Red Hat Enterprise Linux Server' repositories :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5.7"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5.7"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs i386 5.7"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5.7"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5.8"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs i386 5.8"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5.8"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs i386 5Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6.2"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs i386 6.2"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs x86_64 6.2"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs i386 6Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs x86_64 6Server"' :: [ LOG ] :: Duration: 1m 17s :: [ LOG ] :: Assertions: 20 good, 0 bad :: [ PASS ] :: RESULT: Enabling 'Red Hat Enterprise Linux Server' repositories :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: Enabling 'Red Hat CloudForms' repositories :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms Cloud Engine RPMs x86_64 6Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms Cloud Engine RPMs x86_64 6.2"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms System Engine RPMs x86_64 6Server"' :: [ PASS ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms System Engine RPMs x86_64 6.2"' :: [ LOG ] :: Duration: 8s :: [ LOG ] :: Assertions: 4 good, 0 bad :: [ PASS ] :: RESULT: Enabling 'Red Hat CloudForms' repositories :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: Sync and promote 'Red Hat' provider :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 246) :: [ LOG ] :: Duration: 1h 28m 46s :: [ LOG ] :: Assertions: 0 good, 1 bad :: [ FAIL ] :: RESULT: Sync and promote 'Red Hat' provider Actual results: * Katello synchronize command fails, client.log reports (2012-05-16 19:32:29,999 [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">) Expected results: * No failure, as observed in http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2012/05/2333/233390/495957/5814105/TESTOUT.log Additional info: * This automated test is failing when installing CFSE from RHN (classic) * This automated test (same manifest) works fine when installing CFSE from RHSM + I've confirmed that both systems use the same packageset ... perhaps this is unrelated * Logs available for review * /var/log/httpd/* -> http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5814109/tmp-httpd-logs.233391_495958_5814109.tar.gz * /var/log/pulp/* - > http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5814109/tmp-pulp-logs.233391_495958_5814109.tar.gz * /var/log/katello/* - > http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5814109/tmp-katello-logs.233391_495958_5814109.tar.gz
Can I get an 'rpm -qa' from the installed machine from RHN and CDN to see if there are any differences?
(In reply to comment #1) > Can I get an 'rpm -qa' from the installed machine from RHN and CDN to see if > there are any differences? I compared the two package sets, and found no difference. Which was kind of surprising. The good news is that I was able to re-run the 'katello sync' command ... and it completed successfully. That lowers my concern level. Whatever the problem was, it is at least easily resolvable. > # katello -u admin -p admin provider synchronize --name "Red Hat" > Provider [ Red Hat ] synchronized == Suggested workaround == * Retry the synchronize command I still don't know what the error was, maybe some odd timing issue while communicating with RHN/CDN? Hard to know what URL was requested that failed.
In my latest round of tests ... * The sync completes successfully against RHN (classic) * The sync fails against RHSM. However, the failure mode is different this time. I'm starting to think that sync against production systems may just timeout/fail for unknown reasons. The latest failure (see http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5828669/TESTOUT.log) is ... > Progress: [############################################# ] 91.7% > Progress: [############################################# ] 91.7% > Progress: [############################################# ] 91.7% > Progress: [############################################### ] 95.8% > Progress: [############################################### ] 95.8% > Progress: [##################################################] 100.0% > > Provider [ Red Hat ] failed to sync: Traceback (most recent call last): > > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process > retval = method(*args, **kwargs) > > File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 51, in fetchItem > verify_options=self.verify_options, probing=probing, force=force) > > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch > checksum, headers, retryTimes, packages_location) > > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch > checksum, headers, retryTimes, packages_location) > > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 279, in fetch > curl.perform() > > error: (7, "couldn't connect to host")
Created attachment 585392 [details] katello-debug-20120518075000.tar.gz Grinder errors reported .. > # grep ":ERROR:" /var/log/pulp/grinder.log.1 > 2012-05-17 22:14:06,330 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.5.0-4.4.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.5.0-4.4.el6.x86_64.rpm) > 2012-05-17 22:14:06,335 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.5.0-4.4.el6.x86_64.rpm with 1 retry attempts left. > 2012-05-17 22:14:06,346 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.5.0-4.4.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.5.0-4.4.el6.x86_64.rpm) > 2012-05-17 22:14:06,349 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.5.0-4.4.el6.x86_64.rpm with 0 retry attempts left. > 2012-05-17 22:14:06,365 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.5.0-4.4.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.5.0-4.4.el6.x86_64.rpm) > 2012-05-17 22:14:06,369 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:306 Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/grinder/ParallelFetch.py", line 299, in run > result = self.fetcher.fetchItem(itemInfo) > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 82, in __call__ > return self.object(self, *args, **kwargs) > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 267, in __call__ > return self.__call(method, args, kwargs) > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 243, in __call > return self.__rmi(method.name, args, kwargs) > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 136, in __rmi > raise Exception(ex) > Exception: Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process > retval = method(*args, **kwargs) > File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 51, in fetchItem > verify_options=self.verify_options, probing=probing, force=force) > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch > checksum, headers, retryTimes, packages_location) > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch > checksum, headers, retryTimes, packages_location) > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 279, in fetch > curl.perform() > error: (7, "couldn't connect to host") > 2012-05-17 22:14:06,369 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:307 Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process > retval = method(*args, **kwargs) > File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 51, in fetchItem > verify_options=self.verify_options, probing=probing, force=force) > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch > checksum, headers, retryTimes, packages_location) > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch > checksum, headers, retryTimes, packages_location) > File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 279, in fetch > curl.perform() > error: (7, "couldn't connect to host") > 2012-05-17 22:14:07,789 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.6.0-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.6.0-1.el6.x86_64.rpm) > 2012-05-17 22:14:07,793 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.6.0-1.el6.x86_64.rpm with 1 retry attempts left. > 2012-05-17 22:14:07,824 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-gef-3.5.2-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-gef-3.5.2-1.el6.x86_64.rpm) > 2012-05-17 22:14:07,824 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.6.0-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.6.0-1.el6.x86_64.rpm) > 2012-05-17 22:14:07,826 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-gef-3.5.2-1.el6.x86_64.rpm with 1 retry attempts left. > 2012-05-17 22:14:07,826 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.6.0-1.el6.x86_64.rpm with 0 retry attempts left. > 2012-05-17 22:14:07,834 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-gef-3.5.2-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-gef-3.5.2-1.el6.x86_64.rpm) > 2012-05-17 22:14:07,835 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.6.0-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.6.0-1.el6.x86_64.rpm) > 2012-05-17 22:14:07,836 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-gef-3.5.2-1.el6.x86_64.rpm with 0 retry attempts left. > 2012-05-17 22:14:07,841 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:306 Traceback (most recent call last): > 2012-05-17 22:14:07,841 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:307 Traceback (most recent call last): > 2012-05-17 22:14:07,848 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-gef-3.5.2-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-gef-3.5.2-1.el6.x86_64.rpm) > 2012-05-17 22:14:08,354 385:140219108017920: grinder.ParallelFetch:ERROR: ParallelFetch:306 Traceback (most recent call last): > 2012-05-17 22:14:08,355 385:140219108017920: grinder.ParallelFetch:ERROR: ParallelFetch:307 Traceback (most recent call last):
Recreated while testing 1.0.1. For full logs, see https://beaker.engineering.redhat.com/jobs/246228 /var/log/katello/client.log reports > 2012-06-11 13:21:23,758 [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE html> /var/log/pulp/grinder.log indicates the sync is still progressing
/var/log/katello/production.log reports the following. Could just be just another pulp timeout? > [ERROR: 2012-06-11 13:21:23 #19153] exception when talking to a remote client: Pulp::Task: Request Timeout (GET /pulp/api/tasks/?state=archived&state=current&id=a7e46040-b3e6-11e1-a6e3-80c16e62ddb4) RestClient::RequestTimeout: Pulp::Task: Request Timeout (GET /pulp/api/tasks/?state=archived&state=current&id=a7e46040-b3e6-11e1-a6e3-80c16e62ddb4) > /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:174:in `transmit' > /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute' > /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute' > /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:54:in `get' > /usr/share/katello/lib/http_resource.rb:101:in `get' > /usr/share/katello/lib/resources/pulp.rb:500:in `find' > /usr/share/katello/app/models/pulp_task_status.rb:75:in `refresh' > /usr/share/katello/app/models/pulp_task_status.rb:19:in `refresh' > /usr/share/katello/app/controllers/api/tasks_controller.rb:42:in `show' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:150:in `process_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:11:in `process_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:18:in `process_action' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:462:in `_run__866025032__process_action__1251632600__callbacks' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:221:in `_conditional_callback_around_2760' > /usr/share/katello/lib/util/threadsession.rb:79:in `thread_locals' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:220:in `_conditional_callback_around_2760' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:441:in `_run__866025032__process_action__1251632600__callbacks' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `send' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `_run_process_action_callbacks' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `send' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `run_callbacks' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:17:in `process_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:17:in `process_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30:in `process_action' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21:in `instrument' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:29:in `process_action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:119:in `process' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:41:in `process' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:138:in `dispatch' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:178:in `action' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `dispatch' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:27:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:150:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:93:in `recognize' > /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:82:in `optimized_each' > /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:92:in `recognize' > /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:141:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:493:in `call' > /usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:35:in `call' > /usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34:in `catch' > /usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34:in `call' > /usr/lib/ruby/gems/1.8/gems/sass-3.1.4/lib/sass/../sass/plugin/rack.rb:54:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/head.rb:14:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/params_parser.rb:21:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:182:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:149:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302:in `call' > /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:32:in `call' > /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:28:in `cache' > /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:12:in `cache' > /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:31:in `call' > /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:354:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:46:in `call' > /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416:in `_run_call_callbacks' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:44:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/remote_ip.rb:48:in `call' > /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47:in `call' > /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:13:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:34:in `call' > /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:168:in `call' > /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `send' > /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `method_missing' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:52:in `call' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46:in `each' > /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46:in `call' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57:in `process' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine' > /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!' > /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/bin/thin:6 > /usr/share/katello/script/thin:24:in `load' > /usr/share/katello/script/thin:24 > [FATAL: 2012-06-11 13:21:23 #19153] > NoMethodError (private method `gsub' called for nil:NilClass): > app/controllers/api/api_controller.rb:115:in `format_subsys_exception_hash' > app/controllers/api/api_controller.rb:105:in `exception_with_response'
old... closing.