This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 970315 - RFE: spacewalk-repo-sync does not support yum repos containing bz2 compressed xml files
RFE: spacewalk-repo-sync does not support yum repos containing bz2 compressed...
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Server (Show other bugs)
1.9
All Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Michael Mráka
Red Hat Satellite QA List
:
Depends On:
Blocks: space20
  Show dependency treegraph
 
Reported: 2013-06-03 18:20 EDT by Paul Robert Marino
Modified: 2013-08-02 09:13 EDT (History)
3 users (show)

See Also:
Fixed In Version: spacewalk-backend-1.10.43-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-02 09:13:11 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Paul Robert Marino 2013-06-03 18:20:59 EDT
Description of problem:
Scientific Linux's yum repos contain some bz2 compressed xml files, but spacewalk-repo-sync does not currently have the code to handle this.

to be specific http://ftp.scientificlinux.org/linux/scientific/6/x86_64/os/repodata/updateinfo.xml.bz2 is one of the culprits.
as it stands most of their yum repos seem to contain updateinfo.xml file compressed in bz2 format.

it seams like yum already handles bz2 files but the yum plugins for spacewalk do not.

Version-Release number of selected component (if applicable):

In spacewalk 1.9 it causes this error

Repo http://ftp.scientificlinux.org/linux/scientific/6.4/x86_64/os/ has comps file comps-sl6-x86_64.xml.gz.
ERROR: Not a gzipped file
Sync completed.
Total time: 1:46:18

in Spacewalk 1.8 and lower it causes a trace in the taskomatic logs

it also causes the channels repodata to become inconsistent and in some cases non functional.

ng repos for channel: Scientific Linux 6 Updates FastBug (x86_64)
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.manager.satellite.SystemCommandExecu
tor - Error encountered executing (args=[/usr/bin/spacewalk-repo-sync, --channel, scientific6-x86_64-updates-fast, --type, yum])
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.manager.satellite.SystemCommandExecu
tor - Error message from process: Traceback (most recent call last):
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 103, in <module>
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sys.exit(abs(main() or 0))
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 96, in main
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sync.sync()
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 112, in sync
INFO   | jvm 1    | 2013/04/23 00:30:31 |     self.error_msg("ERROR: %s" % e.value)
INFO   | jvm 1    | 2013/04/23 00:30:31 | AttributeError: 'exceptions.IOError' object has no attribute 'value'
INFO   | jvm 1    | 2013/04/23 00:30:31 |
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - Repo URL: http://ftp.scientificlinux.org/linux/scientific/6/x86_64/updates/fastbugs/
INFO   | jvm 1    | 2013/04/23 00:30:31 | Packages in repo:               106
INFO   | jvm 1    | 2013/04/23 00:30:31 | No new packages to sync.
INFO   | jvm 1    | 2013/04/23 00:30:31 |
INFO   | jvm 1    | 2013/04/23 00:30:31 | 2013-04-23 00:30:31,913 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Traceback (most recent call last):
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 103, in <module>
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sys.exit(abs(main() or 0))
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/bin/spacewalk-repo-sync", line 96, in main
INFO   | jvm 1    | 2013/04/23 00:30:31 |     sync.sync()
INFO   | jvm 1    | 2013/04/23 00:30:31 |   File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 112, in sync
INFO   | jvm 1    | 2013/04/23 00:30:31 |     self.error_msg("ERROR: %s" % e.value)
INFO   | jvm 1    | 2013/04/23 00:30:31 | AttributeError: 'exceptions.IOError' object has no attribute 'value'
INFO   | jvm 1    | 2013/04/23 00:30:31 |


How reproducible:

Consistent and easy

Steps to Reproduce:

create a new channel with http://ftp.scientificlinux.org/linux/scientific/6/x86_64/os/ as a repository sync source


Actual results:

files get synced and added to the database
none of the erratas get synced.
in spacewalk 1.8 reposync crashes before it completes resulting in bad and or incomplete file lists in the in yum which block updates and break kickstarts.

Expected results:

all files should get synced.
all erratas should get imported.
and the repodata should get generated properly.


Additional info:
places where the code need to be updated that I've already found are

around line 178 in /usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py

and around 64 of /usr/lib/python2.6/site-packages/spacewalk/satellite_tools/repo_plugins/yum_src.py in class YumUpdateMetadata
Comment 1 Michael Mráka 2013-06-04 07:29:01 EDT
Fixed in spacewalk master by
commit e300b8cd4623122b7b28b445d6647d1832799e82
    970315 - support both gz and bz2 compressed repo files
Comment 2 Paul Robert Marino 2013-06-04 15:20:28 EDT
I manually back ported it to 1.9 in my testing instance and found every thing worked as expected.
The errors were no longer present in the logs after I kicked off a sync, and a few seconds after the sync completed I started receiving errata notification emails.
Comment 3 Tomáš Kašpárek 2013-08-02 09:13:11 EDT
Fix for this bug is present in Spacewalk 2.0, closing this bug as CURRENTRELEASE.

Note You need to log in before you can comment on or make changes to this bug.