Bug 654605

Summary: spacewalk-repo-sync - when some repository is http:// stops on first 404 error
Product: Red Hat Satellite 5 Reporter: Jiri Kastner <jkastner>
Component: Satellite SynchronizationAssignee: Michael Mráka <mmraka>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Korbel <mkorbel>
Severity: medium Docs Contact:
Priority: medium    
Version: 540CC: cperry, mkorbel, tlestach
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-01 21:57:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 924232    

Description Jiri Kastner 2010-11-18 11:39:58 UTC
Description of problem:
spacewalk-repo-sync fails with traceback, when package is not available and there is no more mirror to try. when same repository is accesible via file:// rest of all packages from repodata are tried and sync finish with 'Sync complete'.

Version-Release number of selected component (if applicable):
satellite 5.4

How reproducible:


Steps to Reproduce:
1. using createrepo create repo
2. remove some packages
3. place it on file system
4. place it somewhere accessible via http://xxx/repo
5. create repository via webui with file:///path-defined-in-step-3/
6. create repository via webui with url from step 4
7. create channel via webui
8. connect channel to repository from step 5.
9. run "spacewalk-repo-sync --channel frequently-changed-repo --type yum"
10. some packages from repo are synced, missing packages produces messages: "failure: xxx/yyyyy.rpm from frequently-changed-repo: [Errno 256] No more mirrors to try."
11. change repository settings for same channel to repo from step 6
12. repeat step 9.


  
Actual results:

spacewalk-repo-sync --channel frequently-changed-repo --type yum
Repo http://frequently-changed-repo.com/rpms/ has 107 packages.
1/72 : xxxxxx.noarch
failed to retrieve xxxxxx.noarch.rpm from frequently-changed-repo
error was [Errno 14] HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 72, in ?
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 66, in main
    sync.main()
  File "/usr/share/rhn/satellite_tools/reposync.py", line 103, in main
    self.import_packages(plugin, url)
  File "/usr/share/rhn/satellite_tools/reposync.py", line 201, in import_packages
    os.remove(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found


Expected results:

spacewalk-repo-sync --channel frequently-changed-repo --type yum
Repo http://frequently-changed-repo-fqdn/some-path/ has 10099 packages.
1/1506 : .noarch
"failure: xxxxxx.rpm from frequently-changed-repo: [Errno 256] No more mirrors to try."
2/1506 : yyyyyy.noarch
....
1506/1506 : zzzzzz.noarch
Sync Complete

Additional info:

Comment 3 Michael Mráka 2013-02-21 13:08:42 UTC
The issue has been fixed sometime in the past.

Current spacewalk nightly works properly:

[root@test04-64 repo]# spacewalk-repo-sync -c spacewalk-nightly-server-fedora17-x86_64 -u http://localhost/pub/repo
Repo URL: http://localhost/pub/repo
Packages in repo:               139
Packages already synced:          0
Packages to sync:               139
1/139 : perl-NOCpulse-CLAC-1.9.8-1.fc18-0.noarch
...
22/139 : SatConfig-bootstrap-1.11.5-1.fc18-0.noarch
failure: spacewalk-nightly/SatConfig-bootstrap-1.11.5-1.fc18.noarch.rpm from spacewalk-nightly-server-fedora17-x86_64: [Errno 256] No more mirrors to try.
http://localhost/pub/repo/spacewalk-nightly/SatConfig-bootstrap-1.11.5-1.fc18.noarch.rpm: [Errno 14] HTTP Error 404 - Not Found : http://localhost/pub/repo/spacewalk-nightly/SatConfig-bootstrap-1.11.5-1.fc18.noarch.rpm
23/139 : spacewalk-proxy-redirect-1.9.7-1.fc18-0.noarch
...
Repo http://localhost/pub/repo has 0 errata.
Sync completed.


Package spacewalk-backend-1.9.28-1.

Comment 5 Martin Korbel 2013-09-25 07:15:04 UTC
VERIFIED on spacewalk-backend-2.0.3-18.el6sat (Sat5.6.0-RHEL6-re20130917.0)
REPRODUCED on spacewalk-backend-1.2.13-11.el5sat (Sat 5.4.0)

Steps to reproduce:
1. Download some packages, create rep and create repositories in sattelite.
This reproducer is for Sat5.6.0 (Sat5.4.0 has not channel.software.createRepo, channel.software.disassociateRepo, channel.software.associateRepo). We have to do steps (where are unsupported APIs) using WebUI.
> mkdir /var/www/html/pub/fuse
> cd /var/www/html/pub/fuse
> wget http://download1.rpmfusion.org/free/fedora/releases/19/Everything/i386/os/kmod-xtables-addons-2.3-3.fc19.i686.rpm http://download1.rpmfusion.org/free/fedora/releases/19/Everything/i386/os/kmod-xtables-addons-3.9.5-301.fc19.i686-2.3-3.fc19.i686.rpm
> createrepo .
> rm kmod-xtables-addons-2.3-3.fc19.i686.rpm
> export PASSWD="xxx"
> spacewalk-api --server localhost --user=admin --password=$PASSWD channel.software.createRepo "%session%" "my-url-repo" "YUM" "http://localhost/pub/fuse"
> spacewalk-api --server localhost --user=admin --password=$PASSWD channel.software.createRepo "%session%" "my-file-repo" "YUM" "file:///var/www/html/pub/fuse/"
2. Create custom channel
>python
>>> import xmlrpclib;
>>> client = xmlrpclib.Server('http://localhost/rpc/api', verbose=0);
>>> key = client.auth.login('admin','xxx');
>>> client.channel.software.create(key,"my-channel","my-channel","my-channel","channel-ia32","",{});
3. associate the "my-file-repo" repo and do repo-sync
> spacewalk-api --server localhost --user=admin --password=$PASSWD channel.software.associateRepo "%session%" "my-channel" "my-file-repo" 
> spacewalk-repo-sync --channel "my-channel" --type yum

Sat5.4.0:
Repo file:///var/www/html/pub/fuse/ has 2 packages.
1/2 : kmod-xtables-addons-3.9.5-301.fc19.i686-2.3-3.fc19-0.i686
2/2 : kmod-xtables-addons-2.3-3.fc19-0.i686
failure: kmod-xtables-addons-2.3-3.fc19.i686.rpm from my-channel: [Errno 256] No more mirrors to try.
Sync complete

Sat5.6.0:
Repo URL: file:///var/www/html/pub/fuse/
Packages in repo:                 2
Packages already synced:          0
Packages to sync:                 2
1/2 : kmod-xtables-addons-2.3-3.fc19-0.i686
failure: kmod-xtables-addons-2.3-3.fc19.i686.rpm from my-channel: [Errno 256] No more mirrors to try.
2/2 : kmod-xtables-addons-3.9.5-301.fc19.i686-2.3-3.fc19-0.i686
Linking packages to channel.
Repo file:///var/www/html/pub/fuse/ has 0 errata.
Sync completed.
Total time: 0:00:01


4. change the repo and again repo-sync
> spacewalk-api --server localhost --user=admin --password=$PASSWD channel.software.disassociateRepo "%session%" "my-channel" "my-file-repo" 
> spacewalk-api --server localhost --user=admin --password=$PASSWD channel.software.associateRepo "%session%" "my-channel" "my-url-repo" 
> spacewalk-repo-sync --channel "my-channel" --type yum


Sat5.4.0:
Repo http://localhost/pub/fuse has 2 packages.
1/1 : kmod-xtables-addons-2.3-3.fc19-0.i686
failure: kmod-xtables-addons-2.3-3.fc19.i686.rpm from my-channel: [Errno 256] No more mirrors to try.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 72, in ?
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 66, in main
    sync.main()
  File "/usr/share/rhn/satellite_tools/reposync.py", line 103, in main
    self.import_packages(plugin, url)
  File "/usr/share/rhn/satellite_tools/reposync.py", line 201, in import_packages
    os.remove(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found

Sat5.6.0:
Repo URL: http://localhost/pub/fuse
Packages in repo:                 2
Packages already synced:          1
Packages to sync:                 1
1/1 : kmod-xtables-addons-2.3-3.fc19-0.i686
failure: kmod-xtables-addons-2.3-3.fc19.i686.rpm from my-channel: [Errno 256] No more mirrors to try.
Linking packages to channel.
Repo http://localhost/pub/fuse has 0 errata.
Sync completed.
Total time: 0:00:00

Comment 6 Clifford Perry 2013-10-01 21:57:01 UTC
Satellite 5.6 has been released. This bug was tracked under the release.  

This bug was either VERIFIED or RELEASE_PENDING (re-verified prior shortly
before release). 

Moving to CLOSED CURRENT_RELEASE. 

Text from Upgrade Erratum follows:

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1395.html