Bug 822455 - [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
[ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE HTML PUBLIC "-//...
Status: CLOSED WONTFIX
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Hammer (Show other bugs)
6.0.0
Unspecified Unspecified
unspecified Severity medium (vote)
: Unspecified
: --
Assigned To: Mike McCune
Katello QA List
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-17 07:46 EDT by James Laska
Modified: 2014-03-26 16:36 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-03-26 16:36:32 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
katello-debug-20120518075000.tar.gz (3.48 MB, application/x-gzip)
2012-05-18 07:55 EDT, James Laska
no flags Details

  None (edit)
Description James Laska 2012-05-17 07:46:06 EDT
Description of problem:

While attempting a content sync, the katello CLI eventually fails ...

> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%
> Progress: [                                                  ] 0.0%<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> :: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 246)

/var/log/katello/client.log shows the follow error in more detail:

> 2012-05-16 19:32:29,999 [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

Version-Release number of selected component (if applicable):
 * candlepin-0.5.26-1.el6.src.rpm
 * katello-0.1.311-1.el6_2.src.rpm
 * katello-candlepin-cert-key-pair-1.0-1.src.rpm
 * katello-certs-tools-1.0.4-1.el6.src.rpm
 * katello-cli-0.1.107-1.el6.src.rpm
 * katello-configure-0.1.107-1.el6.src.rpm
 * katello-qpid-broker-key-pair-1.0-1.src.rpm
 * katello-qpid-client-key-pair-1.0-1.src.rpm
 * katello-selinux-0.1.10-1.el6.src.rpm
 * pulp-1.0.4-1.el6.src.rpm
 * redhat-release-server-6Server-6.2.0.3.el6.src.rpm
 * subscription-manager-0.96.18-1.el6_2.src.rpm

How reproducible:
 * Seems to happen on my CFSE beaker tests when katello is installed from RHN
   + Does *NOT* happen using the same MANIFEST, when testing CFSE and installing katello from RHSM (same package) 

Steps to Reproduce:

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Create custom Org to work with
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: Running 'katello -u admin -p admin org create --name redhat'
:: [   PASS   ] :: Running 'katello -u admin -p admin client remember --option org --value redhat'
:: [   LOG    ] :: Duration: 2s
:: [   LOG    ] :: Assertions: 2 good, 0 bad
:: [   PASS   ] :: RESULT: Create custom Org to work with

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Create custom Environments
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: Running 'katello -u admin -p admin environment create --name Dev --prior Library'
:: [   PASS   ] :: Running 'katello -u admin -p admin environment create --name Test --prior Dev'
:: [   PASS   ] :: Running 'katello -u admin -p admin environment create --name Prod --prior Test'
:: [   PASS   ] :: Running 'katello -u admin -p admin client remember --option environment --value Dev'
:: [   PASS   ] :: Running 'katello -u admin -p admin client remember --option env --value Dev'
:: [   LOG    ] :: Duration: 5s
:: [   LOG    ] :: Assertions: 5 good, 0 bad
:: [   PASS   ] :: RESULT: Create custom Environments

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Import manifest
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: Running 'curl -o /tmp/manifest.zip http://file.rdu.redhat.com/jlaska/manifests_access.zip'
:: [   PASS   ] :: Running 'katello -u admin -p admin provider update --name "Red Hat" --url https://cdn.redhat.com'
:: [   PASS   ] :: Running 'katello -u admin -p admin provider import_manifest --name "Red Hat" --file /tmp/manifest.zip'
:: [   LOG    ] :: Duration: 20m 21s
:: [   LOG    ] :: Assertions: 3 good, 0 bad
:: [   PASS   ] :: RESULT: Import manifest

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Enabling 'Red Hat Enterprise Linux Server' repositories
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5.7"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5.7"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs i386 5.7"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5.7"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5.8"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs i386 5.8"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5.8"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs i386 5Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6.2"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs i386 6.2"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs x86_64 6.2"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs i386 6Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat CloudForms Tools for RHEL 6 RPMs x86_64 6Server"'
:: [   LOG    ] :: Duration: 1m 17s
:: [   LOG    ] :: Assertions: 20 good, 0 bad
:: [   PASS   ] :: RESULT: Enabling 'Red Hat Enterprise Linux Server' repositories

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Enabling 'Red Hat CloudForms' repositories
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms Cloud Engine RPMs x86_64 6Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms Cloud Engine RPMs x86_64 6.2"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms System Engine RPMs x86_64 6Server"'
:: [   PASS   ] :: Running 'katello -u admin -p admin repo enable --product "Red Hat CloudForms" --name "Red Hat CloudForms System Engine RPMs x86_64 6.2"'
:: [   LOG    ] :: Duration: 8s
:: [   LOG    ] :: Assertions: 4 good, 0 bad
:: [   PASS   ] :: RESULT: Enabling 'Red Hat CloudForms' repositories

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Sync and promote 'Red Hat' provider
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 246)
:: [   LOG    ] :: Duration: 1h 28m 46s
:: [   LOG    ] :: Assertions: 0 good, 1 bad
:: [   FAIL   ] :: RESULT: Sync and promote 'Red Hat' provider

Actual results:

* Katello synchronize command fails, client.log reports (2012-05-16 19:32:29,999 [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">)

Expected results:

* No failure, as observed in http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2012/05/2333/233390/495957/5814105/TESTOUT.log

Additional info:
 * This automated test is failing when installing CFSE from RHN (classic)
 * This automated test (same manifest) works fine when installing CFSE from RHSM
   + I've confirmed that both systems use the same packageset ... perhaps this is unrelated


 * Logs available for review
   * /var/log/httpd/* -> http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5814109/tmp-httpd-logs.233391_495958_5814109.tar.gz
   * /var/log/pulp/* - > http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5814109/tmp-pulp-logs.233391_495958_5814109.tar.gz
   * /var/log/katello/* - > http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5814109/tmp-katello-logs.233391_495958_5814109.tar.gz
Comment 1 Mike McCune 2012-05-17 10:39:46 EDT
Can I get an 'rpm -qa' from the installed machine from RHN and CDN to see if there are any differences?
Comment 2 James Laska 2012-05-17 10:43:43 EDT
(In reply to comment #1)
> Can I get an 'rpm -qa' from the installed machine from RHN and CDN to see if
> there are any differences?

I compared the two package sets, and found no difference.  Which was kind of surprising.

The good news is that I was able to re-run the 'katello sync' command ... and it completed successfully.  That lowers my concern level.  Whatever the problem was, it is at least easily resolvable.

> # katello -u admin -p admin provider synchronize --name "Red Hat"
> Provider [ Red Hat ] synchronized         

== Suggested workaround ==

 * Retry the synchronize command

I still don't know what the error was, maybe some odd timing issue while communicating with RHN/CDN?  Hard to know what URL was requested that failed.
Comment 3 James Laska 2012-05-18 07:47:56 EDT
In my latest round of tests ...
 * The sync completes successfully against RHN (classic)
 * The sync fails against RHSM.  However, the failure mode is different this time.

I'm starting to think that sync against production systems may just timeout/fail for unknown reasons.  The latest failure (see http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/5828669/TESTOUT.log) is ...

> Progress: [#############################################     ] 91.7%
> Progress: [#############################################     ] 91.7%
> Progress: [#############################################     ] 91.7%
> Progress: [###############################################   ] 95.8%
> Progress: [###############################################   ] 95.8%
> Progress: [##################################################] 100.0%
>                                                                       
> Provider [ Red Hat ] failed to sync: Traceback (most recent call last):
> 
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process
>     retval = method(*args, **kwargs)
> 
>   File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 51, in fetchItem
>     verify_options=self.verify_options, probing=probing, force=force)
> 
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch
>     checksum, headers, retryTimes, packages_location)
> 
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch
>     checksum, headers, retryTimes, packages_location)
> 
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 279, in fetch
>     curl.perform()
> 
> error: (7, "couldn't connect to host")
Comment 4 James Laska 2012-05-18 07:55:13 EDT
Created attachment 585392 [details]
katello-debug-20120518075000.tar.gz

Grinder errors reported ..

> # grep ":ERROR:" /var/log/pulp/grinder.log.1 
> 2012-05-17 22:14:06,330 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.5.0-4.4.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.5.0-4.4.el6.x86_64.rpm)
> 2012-05-17 22:14:06,335 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.5.0-4.4.el6.x86_64.rpm with 1 retry attempts left.
> 2012-05-17 22:14:06,346 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.5.0-4.4.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.5.0-4.4.el6.x86_64.rpm)
> 2012-05-17 22:14:06,349 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.5.0-4.4.el6.x86_64.rpm with 0 retry attempts left.
> 2012-05-17 22:14:06,365 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.5.0-4.4.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.5.0-4.4.el6.x86_64.rpm)
> 2012-05-17 22:14:06,369 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:306 Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/grinder/ParallelFetch.py", line 299, in run
>     result = self.fetcher.fetchItem(itemInfo)
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 82, in __call__
>     return self.object(self, *args, **kwargs)
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 267, in __call__
>     return self.__call(method, args, kwargs)
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 243, in __call
>     return self.__rmi(method.name, args, kwargs)
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 136, in __rmi
>     raise Exception(ex)
> Exception: Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process
>     retval = method(*args, **kwargs)
>   File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 51, in fetchItem
>     verify_options=self.verify_options, probing=probing, force=force)
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch
>     checksum, headers, retryTimes, packages_location)
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch
>     checksum, headers, retryTimes, packages_location)
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 279, in fetch
>     curl.perform()
> error: (7, "couldn't connect to host")
> 2012-05-17 22:14:06,369 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:307 Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process
>     retval = method(*args, **kwargs)
>   File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 51, in fetchItem
>     verify_options=self.verify_options, probing=probing, force=force)
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch
>     checksum, headers, retryTimes, packages_location)
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 344, in fetch
>     checksum, headers, retryTimes, packages_location)
>   File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 279, in fetch
>     curl.perform()
> error: (7, "couldn't connect to host")
> 2012-05-17 22:14:07,789 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.6.0-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.6.0-1.el6.x86_64.rpm)
> 2012-05-17 22:14:07,793 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.6.0-1.el6.x86_64.rpm with 1 retry attempts left.
> 2012-05-17 22:14:07,824 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-gef-3.5.2-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-gef-3.5.2-1.el6.x86_64.rpm)
> 2012-05-17 22:14:07,824 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.6.0-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.6.0-1.el6.x86_64.rpm)
> 2012-05-17 22:14:07,826 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-gef-3.5.2-1.el6.x86_64.rpm with 1 retry attempts left.
> 2012-05-17 22:14:07,826 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-emf-2.6.0-1.el6.x86_64.rpm with 0 retry attempts left.
> 2012-05-17 22:14:07,834 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-gef-3.5.2-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-gef-3.5.2-1.el6.x86_64.rpm)
> 2012-05-17 22:14:07,835 385:140218525017856: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-emf-2.6.0-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-emf-2.6.0-1.el6.x86_64.rpm)
> 2012-05-17 22:14:07,836 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Retrying fetch of: eclipse-gef-3.5.2-1.el6.x86_64.rpm with 0 retry attempts left.
> 2012-05-17 22:14:07,841 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:306 Traceback (most recent call last):
> 2012-05-17 22:14:07,841 385:140218525017856: grinder.ParallelFetch:ERROR: ParallelFetch:307 Traceback (most recent call last):
> 2012-05-17 22:14:07,848 385:140219108017920: grinder.BaseFetch:ERROR: activeobject:160 Caught exception<(7, "couldn't connect to host")> in fetch(eclipse-gef-3.5.2-1.el6.x86_64.rpm, https://cdn.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/Packages/eclipse-gef-3.5.2-1.el6.x86_64.rpm)
> 2012-05-17 22:14:08,354 385:140219108017920: grinder.ParallelFetch:ERROR: ParallelFetch:306 Traceback (most recent call last):
> 2012-05-17 22:14:08,355 385:140219108017920: grinder.ParallelFetch:ERROR: ParallelFetch:307 Traceback (most recent call last):
Comment 6 James Laska 2012-06-11 13:34:46 EDT
Recreated while testing 1.0.1.  For full logs, see https://beaker.engineering.redhat.com/jobs/246228

/var/log/katello/client.log reports
> 2012-06-11 13:21:23,758 [ERROR][MainThread] error() @ base.py:353 - error: <!DOCTYPE html>

/var/log/pulp/grinder.log indicates the sync is still progressing
Comment 7 James Laska 2012-06-11 13:37:49 EDT
/var/log/katello/production.log reports the following.  Could just be just another pulp timeout?

> [ERROR: 2012-06-11 13:21:23 #19153] exception when talking to a remote client: Pulp::Task: Request Timeout  (GET /pulp/api/tasks/?state=archived&state=current&id=a7e46040-b3e6-11e1-a6e3-80c16e62ddb4) RestClient::RequestTimeout: Pulp::Task: Request Timeout  (GET /pulp/api/tasks/?state=archived&state=current&id=a7e46040-b3e6-11e1-a6e3-80c16e62ddb4)
> /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:174:in `transmit'
> /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute'
> /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute'
> /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:54:in `get'
> /usr/share/katello/lib/http_resource.rb:101:in `get'
> /usr/share/katello/lib/resources/pulp.rb:500:in `find'
> /usr/share/katello/app/models/pulp_task_status.rb:75:in `refresh'
> /usr/share/katello/app/models/pulp_task_status.rb:19:in `refresh'
> /usr/share/katello/app/controllers/api/tasks_controller.rb:42:in `show'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:150:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:11:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:18:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:462:in `_run__866025032__process_action__1251632600__callbacks'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:221:in `_conditional_callback_around_2760'
> /usr/share/katello/lib/util/threadsession.rb:79:in `thread_locals'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:220:in `_conditional_callback_around_2760'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:441:in `_run__866025032__process_action__1251632600__callbacks'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `send'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `_run_process_action_callbacks'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `send'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `run_callbacks'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:17:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:17:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:119:in `process'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:41:in `process'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:138:in `dispatch'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:178:in `action'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `dispatch'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:27:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:150:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:93:in `recognize'
> /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:82:in `optimized_each'
> /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:92:in `recognize'
> /usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:141:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:493:in `call'
> /usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:35:in `call'
> /usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34:in `catch'
> /usr/lib/ruby/gems/1.8/gems/warden-1.0.5/lib/warden/manager.rb:34:in `call'
> /usr/lib/ruby/gems/1.8/gems/sass-3.1.4/lib/sass/../sass/plugin/rack.rb:54:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/head.rb:14:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:182:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:149:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302:in `call'
> /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:32:in `call'
> /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:28:in `cache'
> /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:12:in `cache'
> /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/query_cache.rb:31:in `call'
> /usr/lib/ruby/gems/1.8/gems/activerecord-3.0.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:354:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:46:in `call'
> /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416:in `_run_call_callbacks'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:44:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/remote_ip.rb:48:in `call'
> /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47:in `call'
> /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:13:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:34:in `call'
> /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:168:in `call'
> /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `send'
> /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `method_missing'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:52:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46:in `each'
> /usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/urlmap.rb:46:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:57:in `process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.11/bin/thin:6
> /usr/share/katello/script/thin:24:in `load'
> /usr/share/katello/script/thin:24
> [FATAL: 2012-06-11 13:21:23 #19153] 
> NoMethodError (private method `gsub' called for nil:NilClass):
>   app/controllers/api/api_controller.rb:115:in `format_subsys_exception_hash'
>   app/controllers/api/api_controller.rb:105:in `exception_with_response'
Comment 11 Bryan Kearney 2014-03-26 16:36:32 EDT
old... closing.

Note You need to log in before you can comment on or make changes to this bug.