Bug 970315 - RFE: spacewalk-repo-sync does not support yum repos containing bz2 compressed xml files
Summary: RFE: spacewalk-repo-sync does not support yum repos containing bz2 compressed...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Spacewalk
Classification: Community
Component: Server
Version: 1.9
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Michael Mráka
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks: space20
TreeView+ depends on / blocked
 
Reported: 2013-06-03 22:20 UTC by Paul Robert Marino
Modified: 2013-08-02 13:13 UTC (History)
3 users (show)

Fixed In Version: spacewalk-backend-1.10.43-1
Clone Of:
Environment:
Last Closed: 2013-08-02 13:13:11 UTC
Embargoed:


Attachments (Terms of Use)

Description Paul Robert Marino 2013-06-03 22:20:59 UTC
Description of problem:
Scientific Linux's yum repos contain some bz2 compressed xml files, but spacewalk-repo-sync does not currently have the code to handle this.

to be specific http://ftp.scientificlinux.org/linux/scientific/6/x86_64/os/repodata/updateinfo.xml.bz2 is one of the culprits.
as it stands most of their yum repos seem to contain updateinfo.xml file compressed in bz2 format.

it seams like yum already handles bz2 files but the yum plugins for spacewalk do not.

Version-Release number of selected component (if applicable):

In spacewalk 1.9 it causes this error

Repo http://ftp.scientificlinux.org/linux/scientific/6.4/x86_64/os/ has comps file comps-sl6-x86_64.xml.gz.
ERROR: Not a gzipped file
Sync completed.
Total time: 1:46:18

in Spacewalk 1.8 and lower it causes a trace in the taskomatic logs

it also causes the channels repodata to become inconsistent and in some cases non functional.

ng repos for channel: Scientific Linux 6 Updates FastBug (x86_64)
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.manager.satellite.SystemCommandExecu
tor - Error encountered executing (args=[/usr/bin/spacewalk-repo-sync, --channel, scientific6-x86_64-updates-fast, --type, yum])
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.manager.satellite.SystemCommandExecu
tor - Error message from process: Traceback (most recent call last):
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 103, in <module>
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sys.exit(abs(main() or 0))
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 96, in main
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sync.sync()
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 112, in sync
INFO   | jvm 1    | 2013/04/23 00:30:31 |     self.error_msg("ERROR: %s" % e.value)
INFO   | jvm 1    | 2013/04/23 00:30:31 | AttributeError: 'exceptions.IOError' object has no attribute 'value'
INFO   | jvm 1    | 2013/04/23 00:30:31 |
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - Repo URL: http://ftp.scientificlinux.org/linux/scientific/6/x86_64/updates/fastbugs/
INFO   | jvm 1    | 2013/04/23 00:30:31 | Packages in repo:               106
INFO   | jvm 1    | 2013/04/23 00:30:31 | No new packages to sync.
INFO   | jvm 1    | 2013/04/23 00:30:31 |
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Traceback (most recent call last):
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 103, in <module>
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sys.exit(abs(main() or 0))
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 96, in main
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sync.sync()
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 112, in sync
INFO   | jvm 1    | 2013/04/23 00:30:31 |     self.error_msg("ERROR: %s" % e.value)
INFO   | jvm 1    | 2013/04/23 00:30:31 | AttributeError: 'exceptions.IOError' object has no attribute 'value'
INFO   | jvm 1    | 2013/04/23 00:30:31 |


How reproducible:

Consistent and easy

Steps to Reproduce:

create a new channel with http://ftp.scientificlinux.org/linux/scientific/6/x86_64/os/ as a repository sync source


Actual results:

files get synced and added to the database
none of the erratas get synced.
in spacewalk 1.8 reposync crashes before it completes resulting in bad and or incomplete file lists in the in yum which block updates and break kickstarts.

Expected results:

all files should get synced.
all erratas should get imported.
and the repodata should get generated properly.


Additional info:
places where the code need to be updated that I've already found are

around line 178 in /usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py

and around 64 of /usr/lib/python2.6/site-packages/spacewalk/satellite_tools/repo_plugins/yum_src.py in class YumUpdateMetadata

Comment 1 Michael Mráka 2013-06-04 11:29:01 UTC
Fixed in spacewalk master by
commit e300b8cd4623122b7b28b445d6647d1832799e82
    970315 - support both gz and bz2 compressed repo files

Comment 2 Paul Robert Marino 2013-06-04 19:20:28 UTC
I manually back ported it to 1.9 in my testing instance and found every thing worked as expected.
The errors were no longer present in the logs after I kicked off a sync, and a few seconds after the sync completed I started receiving errata notification emails.

Comment 3 Tomáš Kašpárek 2013-08-02 13:13:11 UTC
Fix for this bug is present in Spacewalk 2.0, closing this bug as CURRENTRELEASE.


Note You need to log in before you can comment on or make changes to this bug.