Bug 613585 - inter satellite-sync fails with SYNC ERROR (500)
Summary: inter satellite-sync fails with SYNC ERROR (500)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Spacewalk
Classification: Community
Component: Server
Version: 1.0
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Justin Sherrill
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks: space11
TreeView+ depends on / blocked
 
Reported: 2010-07-12 10:57 UTC by Christoph Sievers
Modified: 2010-08-19 08:24 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-08-19 08:24:16 UTC
Embargoed:


Attachments (Terms of Use)

Description Christoph Sievers 2010-07-12 10:57:26 UTC
Description of problem:

trying list-channels with satellite-sync fails with http 500

Version-Release number of selected component (if applicable):

since spacewalk 1.0

How reproducible:

Steps to Reproduce:
1. make sure iss is allowed for the asking system then
2. satellite-sync --iss-parent=masterspacewalk --list-channels --no-ssl
  
Actual results:

[root@spacewalk ~]# satellite-sync --iss-parent=masterspacewalk --list-channels --no-ssl
12:51:04 Red Hat Network Satellite - live synchronization
12:51:04    url: http://masterspacewalk
12:51:04    debug/output level: 1
12:51:04    db:  spacewalk/<password>@xe
12:51:04 
12:51:04 Retrieving / parsing channel-families data
+++ sending log as an email +++

SYNC ERROR:

(Check logs/email for potentially more detail)

ERROR: server.dump.channel_families('<the systemid>',): <ProtocolError for masterspacewalk /SAT-DUMP-INTERNAL: 500 Internal Server Error>

Expected results:

list of software channels

Additional info:

I found some errors within the apache error log. One execution of satellite-sync leads to the following errors:

[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:44 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonHeaderParserHandler satellite_exporter.satexport::headerParserHandler: TypeError: __init__() should return None
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: Traceback (most recent call last):
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 287, in HandlerDispatch\n    log=debug)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/lib64/python2.4/site-packages/mod_python/apache.py", line 464, in import_module\n    module = imp.load_module(mname, f, p, d)
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler:   File "/usr/share/rhn/satellite_exporter/satexport.py", line 249, in ?\n    headerParserHandler = HandlerWrap("headerParserHandler")
[Mon Jul 12 11:31:45 2010] [error] [client slavespacewalk] PythonCleanupHandler satellite_exporter.satexport::cleanupHandler: TypeError: __init__() should return None

rgds
Christoph

Comment 1 Christoph Sievers 2010-07-12 13:24:32 UTC
With jsherril's help I applied

http://git.fedorahosted.org/git/?p=spacewalk.git;a=blobdiff;f=backend/satellite_exporter/satexport.py;h=58ea591a4b56b2241667fc897cc3c98191437eda;hp=8fca481c647caf3e3e70195f5389f032569a68b3;hb=b6a487af6473d73a768f99739cbe0b5a73ad5108;hpb=c2572de980b308e7ba2250f644fc2350374bc7df


which is

--- a/backend/satellite_exporter/satexport.py
+++ b/backend/satellite_exporter/satexport.py
@@ -240,10 +240,10 @@ class ApacheServer(BaseApacheServer):
 class FunctionRetrievalError(Exception):
     pass
 
-apache_server = ApacheServer()
-class HandlerWrap:
-    def __init__(self, name, init=0):
-        return getattr(apache_server, name)
+from server import apacheServer
+class HandlerWrap(apacheServer.HandlerWrap):
+    def get_handler_factory(self, req):
+        return ApacheServer
 
 ### Instantiate external entry points:
 headerParserHandler = HandlerWrap("headerParserHandler")


I now get the following Errors in the apache error log

Exception reported from spacewalk
Time: Mon Jul 12 15:16:34 2010
Exception type exceptions.KeyError
Exception while handling function dump.channel_families
Request object information:
URI: /SAT-DUMP-INTERNAL
Remote Host: x.x.x.x
Server Name: masterspacewalk:0
Headers passed in:
	Accept-Encoding: identity
	Content-Length: 2175
	Host: masterspacewalk
	content-type: text/xml
	user-agent: satellite-sync/0.1
	x-client-version: 1
	x-info: RPC Processor (C) Red Hat, Inc (version $Revision$)
	x-rhn-satellite-xml-dump-version: 3.5
	x-rhn-transport-capability: follow-redirects=3
	x-transport-info: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)
Extra information about this error:
Response sent back to the caller:
While running 'dump.channel_families': caught
exceptions.KeyError : 'non_auth_dumper'



Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/server/apacheRequest.py", line 118, in call_function
    func = self.method_ref(method)
  File "/usr/share/rhn/server/apacheRequest.py", line 413, in method_ref
    classes = self.servers[self.server]
KeyError: 'non_auth_dumper'

Comment 2 Michael Mráka 2010-07-12 13:38:19 UTC
Reassigning to Justin.

Comment 3 Justin Sherrill 2010-07-12 21:22:04 UTC
I think this should resolve the issue:

9c1d1f8ee656afe2aadbc6a0dd5a9ddcc20ecce3

Packages are available here if you would like to test (Please do!):

el5 packages:
http://koji.spacewalkproject.org/koji/buildinfo?buildID=18157

just download all those and rpm -Uvh  them.

-Justin

Comment 4 Christoph Sievers 2010-07-13 07:02:58 UTC
Hi,

seems to work now. Thank you for this good job.

Christoph

Comment 5 Christoph Sievers 2010-07-13 08:22:31 UTC
Sorry I think there is still a problem left

I can start the sync and it does a lot of work but then in the end..

<snip>
0:03:47     2781/2783 Fetch successful: policycoreutils-newrole-1.33.12-14.6.el5.x86_64.rpm (65630 bytes)
10:03:47     2782/2783 Fetch successful: qcairo-devel-1.8.7.1-3.el5.i386.rpm (170561 bytes)
10:03:47     2783/2783 Fetch successful: oprofile-devel-0.9.4-11.el5.i386.rpm (168416 bytes)
10:03:47 Processing rpm packages complete
10:03:47 
10:03:47 Downloading package metadata
10:03:48    Retrieving / parsing *relevant* package metadata: oracle-el5-x86_64 (4263)
10:03:48    * WARNING: this may be a slow process.
            ________________________________________
Downloading:######################################## - complete
10:08:46 Environment changed, trying again...
10:08:46 Repeated failures
    Import complete:
        Begin time: Tue Jul 13 09:23:05 2010
        End time:   Tue Jul 13 10:08:46 2010
        Elapsed:    0 hours, 45 minutes, 41 seconds
          


Looks like there were errors..
Packages do not come up in the channel

tried again but
same result

10:14:58    Retrieving / parsing *relevant* package metadata: oracle-el5-x86_64 (4263)
10:14:58    * WARNING: this may be a slow process.
            ________________________________________
Downloading:######################################## - complete
10:19:55 Environment changed, trying again...
10:19:55 Repeated failures
    Import complete:
        Begin time: Tue Jul 13 10:09:47 2010
        End time:   Tue Jul 13 10:19:55 2010
        Elapsed:    0 hours, 10 minutes, 7 seconds


rgds
Christoph

Comment 6 Christoph Sievers 2010-07-13 08:54:16 UTC
tried with debug

satellite-sync --iss-parent=masterspacewalk --no-ssl -c oracle-el5-x86_64 --debug-level=4

/var/log/rhn/rhn_sat_export_internal.log contains a bunch of lines

2010/07/13 10:50:47 +02:00 18891 0.0.0.0: satellite_exporter/satexport._validate_version('Server version', 3.5, 'Client version', '3.5')
2010/07/13 10:50:47 +02:00 18891 0.0.0.0: disk_dumper/dumper.__init__('Use database cache', 0)
2010/07/13 10:50:53 +02:00 18572 0.0.0.0: satellite_exporter/satexport._validate_version('Server version', 3.5, 'Client version', '3.5')
2010/07/13 10:50:53 +02:00 18572 0.0.0.0: disk_dumper/dumper.__init__('Use database cache', 0)
2010/07/13 10:51:00 +02:00 18856 0.0.0.0: satellite_exporter/satexport._validate_version('Server version', 3.5, 'Client version', '3.5')
2010/07/13 10:51:00 +02:00 18856 0.0.0.0: disk_dumper/dumper.__init__('Use database cache', 0)

dont see any errors within catalina.out or httpd logs..

Comment 7 Christoph Sievers 2010-07-14 08:24:45 UTC
I exported the channel with rhn-satellite-exporter and imported that on the other spacewalk.

It is NOT getting those Enviroment changed exceptions.
 >Environment changed, trying again...
 >Repeated failures
and

THAT import was successful so maybe this is a server side problem?

satellite-sync log:

[root@spacewalk oel5]# satellite-sync -m /root/oel5/ -c oracle-el5-x86_64
09:58:11 Red Hat Network Satellite - file-system synchronization
09:58:11    mp:  /root/oel5
09:58:11    db:  spacewalk/<password>@xe
09:58:11
09:58:11 Retrieving / parsing channel-families data
09:58:12 channel-families data complete
09:58:12
09:58:12 Retrieving / parsing arches data
09:58:12 arches data complete
09:58:12
09:58:12 Retrieving / parsing additional arches data
09:58:12 additional arches data complete
09:58:12
09:58:12 Retrieving / parsing channel data
09:58:12    p = previously imported/synced channel
09:58:12    . = channel not yet imported/synced
09:58:12    base-channels:
09:58:12       . oracle-el5-x86_64                        4263
09:58:12
09:58:13 Channel data complete
09:58:13
09:58:13 Retrieving / parsing blacklists data
09:58:13 blacklists data complete
09:58:13
09:58:13 Retrieving / parsing product names data
09:58:13
09:58:13 Retrieving short package metadata (used for indexing)
09:58:13    Retrieving / parsing short package metadata: oracle-el5-x86_64 (4263)
09:58:21 Diffing package metadata (what's missing locally?): oracle-el5-x86_64
            ________________________________________
Diffing:    ######################################## - complete
10:00:29
10:00:29 Downloading rpm packages
10:00:29    Fetching any missing RPMs: oracle-el5-x86_64 (NONE MISSING)
10:00:29 Processing rpm packages complete
10:00:29
10:00:29 Downloading package metadata
10:00:30    Retrieving / parsing *relevant* package metadata: oracle-el5-x86_64 (NONE RELEVANT)
10:00:30
10:00:30 Downloading errata data
10:00:31    Retrieving / parsing errata data: oracle-el5-x86_64 (0)
10:00:31 Downloading errata data complete
10:00:31
10:00:31 Downloading kickstartable trees metadata
10:00:31    Retrieving / parsing kickstart data: oracle-el5-x86_64 (NONE RELEVANT)
10:00:31
10:00:31 Downloading kickstartable trees files
10:00:31    Retrieving / parsing kickstart tree files: oracle-el5-x86_64 (NONE RELEVANT)
10:00:31
10:00:31 Importing package metadata
10:00:31    Importing *relevant* package metadata: oracle-el5-x86_64 (4263)
10:00:31    * WARNING: this may be a slow process.
            ________________________________________
Importing:  ######################################## - complete
10:19:55 
10:19:55 Linking packages to channels
10:20:30 
10:20:30 Importing channel errata
10:20:30    Importing *relevant* errata: oracle-el5-x86_64 (NONE RELEVANT)
10:20:30    No new kickstartable tree to import
    Import complete:
        Begin time: Wed Jul 14 09:58:11 2010
        End time:   Wed Jul 14 10:20:30 2010
        Elapsed:    0 hours, 22 minutes, 18 seconds

Comment 8 Milan Zázrivec 2010-08-19 08:24:16 UTC
Spacewalk 1.1 has been released.


Note You need to log in before you can comment on or make changes to this bug.