Bug 613585

Summary: inter satellite-sync fails with SYNC ERROR (500)
Product: [Community] Spacewalk Reporter: Christoph Sievers <christoph.sievers>
Component: ServerAssignee: Justin Sherrill <jsherril>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: medium Docs Contact:
Priority: low    
Version: 1.0   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-08-19 08:24:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 585232    

Description Christoph Sievers 2010-07-12 10:57:26 UTC
Description of problem:

trying list-channels with satellite-sync fails with http 500

Version-Release number of selected component (if applicable):

since spacewalk 1.0

How reproducible:

Steps to Reproduce:
1. make sure iss is allowed for the asking system then
2. satellite-sync --iss-parent=masterspacewalk --list-channels --no-ssl
  
Actual results:

[root@spacewalk ~]# satellite-sync --iss-parent=masterspacewalk --list-channels --no-ssl
12:51:04 Red Hat Network Satellite - live synchronization
12:51:04    url: http://masterspacewalk
12:51:04    debug/output level: 1
12:51:04    db:  spacewalk/<password>@xe
12:51:04 
12:51:04 Retrieving / parsing channel-families data
+++ sending log as an email +++

SYNC ERROR:

(Check logs/email for potentially more detail)

ERROR: server.dump.channel_families('<the systemid>',): <ProtocolError for masterspacewalk /SAT-DUMP-INTERNAL: 500 Internal Server Error>

Expected results:

list of software channels

Additional info:

I found some errors within the apache error log. One execution of satellite-sync leads to the following errors:

[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None

rgds
Christoph

Comment 1 Christoph Sievers 2010-07-12 13:24:32 UTC
With jsherril's help I applied

http://git.fedorahosted.org/git/?p=spacewalk.git;a=blobdiff;f=backend/satellite_exporter/satexport.py;h=58ea591a4b56b2241667fc897cc3c98191437eda;hp=8fca481c647caf3e3e70195f5389f032569a68b3;hb=b6a487af6473d73a768f99739cbe0b5a73ad5108;hpb=c2572de980b308e7ba2250f644fc2350374bc7df


which is

--- a/backend/satellite_exporter/satexport.py
+++ b/backend/satellite_exporter/satexport.py
@@ -240,10 +240,10 @@ class ApacheServer(BaseApacheServer):
 class FunctionRetrievalError(Exception):
     pass
 
-apache_server = ApacheServer()
-class HandlerWrap:
-    def __init__(self, name, init=0):
-        return getattr(apache_server, name)
+from server import apacheServer
+class HandlerWrap(apacheServer.HandlerWrap):
+    def get_handler_factory(self, req):
+        return ApacheServer
 
 ### Instantiate external entry points:
 headerParserHandler = HandlerWrap("headerParserHandler")


I now get the following Errors in the apache error log

Exception reported from spacewalk
Time: Mon Jul 12 15:16:34 2010
Exception type exceptions.KeyError
Exception while handling function dump.channel_families
Request object information:
URI: /SAT-DUMP-INTERNAL
Remote Host: x.x.x.x
Server Name: masterspacewalk:0
Headers passed in:
	Accept-Encoding: identity
	Content-Length: 2175
	Host: masterspacewalk
	content-type: text/xml
	user-agent: satellite-sync/0.1
	x-client-version: 1
	x-info: RPC Processor (C) Red Hat, Inc (version $Revision$)
	x-rhn-satellite-xml-dump-version: 3.5
	x-rhn-transport-capability: follow-redirects=3
	x-transport-info: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)
Extra information about this error:
Response sent back to the caller:
While running 'dump.channel_families': caught
exceptions.KeyError : 'non_auth_dumper'



Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/server/apacheRequest.py", line 118, in call_function
    func = self.method_ref(method)
  File "/usr/share/rhn/server/apacheRequest.py", line 413, in method_ref
    classes = self.servers[self.server]
KeyError: 'non_auth_dumper'

Comment 2 Michael Mráka 2010-07-12 13:38:19 UTC
Reassigning to Justin.

Comment 3 Justin Sherrill 2010-07-12 21:22:04 UTC
I think this should resolve the issue:

9c1d1f8ee656afe2aadbc6a0dd5a9ddcc20ecce3

Packages are available here if you would like to test (Please do!):

el5 packages:
http://koji.spacewalkproject.org/koji/buildinfo?buildID=18157

just download all those and rpm -Uvh  them.

-Justin

Comment 4 Christoph Sievers 2010-07-13 07:02:58 UTC
Hi,

seems to work now. Thank you for this good job.

Christoph

Comment 5 Christoph Sievers 2010-07-13 08:22:31 UTC
Sorry I think there is still a problem left

I can start the sync and it does a lot of work but then in the end..

<snip>
0:03:47     2781/2783 Fetch successful: policycoreutils-newrole-1.33.12-14.6.el5.x86_64.rpm (65630 bytes)
10:03:47     2782/2783 Fetch successful: qcairo-devel-1.8.7.1-3.el5.i386.rpm (170561 bytes)
10:03:47     2783/2783 Fetch successful: oprofile-devel-0.9.4-11.el5.i386.rpm (168416 bytes)
10:03:47 Processing rpm packages complete
10:03:47 
10:03:47 Downloading package metadata
10:03:48    Retrieving / parsing *relevant* package metadata: oracle-el5-x86_64 (4263)
10:03:48    * WARNING: this may be a slow process.
            ________________________________________
Downloading:######################################## - complete
10:08:46 Environment changed, trying again...
10:08:46 Repeated failures
    Import complete:
        Begin time: Tue Jul 13 09:23:05 2010
        End time:   Tue Jul 13 10:08:46 2010
        Elapsed:    0 hours, 45 minutes, 41 seconds
          


Looks like there were errors..
Packages do not come up in the channel

tried again but
same result

10:14:58    Retrieving / parsing *relevant* package metadata: oracle-el5-x86_64 (4263)
10:14:58    * WARNING: this may be a slow process.
            ________________________________________
Downloading:######################################## - complete
10:19:55 Environment changed, trying again...
10:19:55 Repeated failures
    Import complete:
        Begin time: Tue Jul 13 10:09:47 2010
        End time:   Tue Jul 13 10:19:55 2010
        Elapsed:    0 hours, 10 minutes, 7 seconds


rgds
Christoph

Comment 6 Christoph Sievers 2010-07-13 08:54:16 UTC
tried with debug

satellite-sync --iss-parent=masterspacewalk --no-ssl -c oracle-el5-x86_64 --debug-level=4

/var/log/rhn/rhn_sat_export_internal.log contains a bunch of lines

2010/07/13 10:50:47 +02:00 18891 0.0.0.0: satellite_exporter/satexport._validate_version('Server version', 3.5, 'Client version', '3.5')
2010/07/13 10:50:47 +02:00 18891 0.0.0.0: disk_dumper/dumper.__init__('Use database cache', 0)
2010/07/13 10:50:53 +02:00 18572 0.0.0.0: satellite_exporter/satexport._validate_version('Server version', 3.5, 'Client version', '3.5')
2010/07/13 10:50:53 +02:00 18572 0.0.0.0: disk_dumper/dumper.__init__('Use database cache', 0)
2010/07/13 10:51:00 +02:00 18856 0.0.0.0: satellite_exporter/satexport._validate_version('Server version', 3.5, 'Client version', '3.5')
2010/07/13 10:51:00 +02:00 18856 0.0.0.0: disk_dumper/dumper.__init__('Use database cache', 0)

dont see any errors within catalina.out or httpd logs..

Comment 7 Christoph Sievers 2010-07-14 08:24:45 UTC
I exported the channel with rhn-satellite-exporter and imported that on the other spacewalk.

It is NOT getting those Enviroment changed exceptions.
 >Environment changed, trying again...
 >Repeated failures
and

THAT import was successful so maybe this is a server side problem?

satellite-sync log:

[root@spacewalk oel5]# satellite-sync -m /root/oel5/ -c oracle-el5-x86_64
09:58:11 Red Hat Network Satellite - file-system synchronization
09:58:11    mp:  /root/oel5
09:58:11    db:  spacewalk/<password>@xe
09:58:11
09:58:11 Retrieving / parsing channel-families data
09:58:12 channel-families data complete
09:58:12
09:58:12 Retrieving / parsing arches data
09:58:12 arches data complete
09:58:12
09:58:12 Retrieving / parsing additional arches data
09:58:12 additional arches data complete
09:58:12
09:58:12 Retrieving / parsing channel data
09:58:12    p = previously imported/synced channel
09:58:12    . = channel not yet imported/synced
09:58:12    base-channels:
09:58:12       . oracle-el5-x86_64                        4263
09:58:12
09:58:13 Channel data complete
09:58:13
09:58:13 Retrieving / parsing blacklists data
09:58:13 blacklists data complete
09:58:13
09:58:13 Retrieving / parsing product names data
09:58:13
09:58:13 Retrieving short package metadata (used for indexing)
09:58:13    Retrieving / parsing short package metadata: oracle-el5-x86_64 (4263)
09:58:21 Diffing package metadata (what's missing locally?): oracle-el5-x86_64
            ________________________________________
Diffing:    ######################################## - complete
10:00:29
10:00:29 Downloading rpm packages
10:00:29    Fetching any missing RPMs: oracle-el5-x86_64 (NONE MISSING)
10:00:29 Processing rpm packages complete
10:00:29
10:00:29 Downloading package metadata
10:00:30    Retrieving / parsing *relevant* package metadata: oracle-el5-x86_64 (NONE RELEVANT)
10:00:30
10:00:30 Downloading errata data
10:00:31    Retrieving / parsing errata data: oracle-el5-x86_64 (0)
10:00:31 Downloading errata data complete
10:00:31
10:00:31 Downloading kickstartable trees metadata
10:00:31    Retrieving / parsing kickstart data: oracle-el5-x86_64 (NONE RELEVANT)
10:00:31
10:00:31 Downloading kickstartable trees files
10:00:31    Retrieving / parsing kickstart tree files: oracle-el5-x86_64 (NONE RELEVANT)
10:00:31
10:00:31 Importing package metadata
10:00:31    Importing *relevant* package metadata: oracle-el5-x86_64 (4263)
10:00:31    * WARNING: this may be a slow process.
            ________________________________________
Importing:  ######################################## - complete
10:19:55 
10:19:55 Linking packages to channels
10:20:30 
10:20:30 Importing channel errata
10:20:30    Importing *relevant* errata: oracle-el5-x86_64 (NONE RELEVANT)
10:20:30    No new kickstartable tree to import
    Import complete:
        Begin time: Wed Jul 14 09:58:11 2010
        End time:   Wed Jul 14 10:20:30 2010
        Elapsed:    0 hours, 22 minutes, 18 seconds

Comment 8 Milan Zázrivec 2010-08-19 08:24:16 UTC
Spacewalk 1.1 has been released.