Bug 455728 - yumdownloader doesn't iterate across multiple repos with matching architectures
yumdownloader doesn't iterate across multiple repos with matching architectures
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: yum-utils (Show other bugs)
9
All Linux
low Severity low
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-07-17 09:18 EDT by Matt Hicks
Modified: 2008-08-07 11:49 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-08-07 11:49:29 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Patch to handle RPM resolution across multiple repos (1.27 KB, text/plain)
2008-07-17 09:18 EDT, Matt Hicks
no flags Details
Repository configuration generated from Revisor (840 bytes, text/plain)
2008-07-17 10:10 EDT, Matt Hicks
no flags Details

  None (edit)
Description Matt Hicks 2008-07-17 09:18:19 EDT
Description of problem:
We found that when multiple repos with the same architecture are present in the
yum configuration, yumdownloader only checks the first repo and then fails if
the desired RPM doesn't exist.  For example, if you have repo1-i386 and
repo2-i386 with repo2-i386 containing crazy.rpm, yumdownloader will error out
when it can't find crazy.rpm in repo1-i386.  We have attached a patch that
catches the yum error and continues the iterate through the rest of the
repositories.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Matt Hicks 2008-07-17 09:18:19 EDT
Created attachment 312049 [details]
Patch to handle RPM resolution across multiple repos
Comment 2 seth vidal 2008-07-17 09:44:48 EDT
Maybe I'm not sure I'm following your issue here. But this patch is catching the
issue where repo1 no longer has any mirrors with the file but the preferred file
is in repo1.

You seem to be treating repos like mirrors.

Could you provide me with the content of the .repo files you have where you are
seeing this case?
Comment 3 Matt Hicks 2008-07-17 10:10:06 EDT
Created attachment 312050 [details]
Repository configuration generated from Revisor
Comment 4 Matt Hicks 2008-07-17 10:12:54 EDT
I uploaded the repository configuration generated from Revisor.  Right now, we
are having to fake out the ftp.redhat.com internally until we have pushed the
content there, but inside Red Hat's network, you can replace
  http://ftp.redhat.com/pub/redhat/genome/yum
with
  http://oss1-repo.usersys.redhat.com/cobbler/repo_mirror/

The command that Revisor was calling that failed was:
  yumdownloader -c ~/test.repo anaconda-runtime
Comment 5 seth vidal 2008-07-17 10:18:20 EDT
So you have multiple repos which have overlapping sets of pkgs? eg: you have the
pkg 'foo' in more than one repo?

Comment 6 Jeroen van Meeuwen 2008-07-17 10:23:59 EDT
Note that the anaconda-runtime package is available in multiple locations
(multiple repositories to be exact), anaconda-extrarepo-1 and anaconda-extrarepo-3
Comment 7 Jeroen van Meeuwen 2008-07-17 10:27:09 EDT
(In reply to comment #5)
> So you have multiple repos which have overlapping sets of pkgs? eg: you have the
> pkg 'foo' in more than one repo?
> 

Yes, but one has a more recent nevra then the other. In this case,
anaconda-extrarepo-3 has the most recent (anaconda-11.4.0.82-4),
anaconda-extrarepo-1 has anaconda-11.4.0.82-1
Comment 8 Matt Hicks 2008-07-17 10:28:16 EDT
No, that is why we thought it was a bug.  When we ran the command:
  yumdownloader -c ~/test.repo anaconda-runtime

It would check 'anaconda-extrarepo-2' for the anaconda-runtime RPM (which is
doesn't contain.  In the packages list returned in yumdownloader:
  packages =  self.pkgSack.searchNevra(n,e,v,r,a)

the package 'anaconda-runtime' is listed twice, one linking 'package.repoid' to
'anaconda-extrarepo-2' and one linking it to 'anaconda-extrarepo-4'.  The
'anaconda-extrarepo-4' is the correct repo, but it never gets to that point
because it breaks out of the loop when the RPM doesn't exist in
'anaconda-extrarepo-2'.
Comment 9 Jeroen van Meeuwen 2008-07-17 10:51:45 EDT
(In reply to comment #8)
> 'anaconda-extrarepo-2' and one linking it to 'anaconda-extrarepo-4'.  The
> 'anaconda-extrarepo-4' is the correct repo,

Uch, why did I type 'anaconda-extrarepo-3' twice? I meant 4, sorry for the
confusion.

Comment 10 seth vidal 2008-07-17 11:06:03 EDT
what ver of yum and yum-utils are you running. I set up a yum.conf like yours
(excluding the first repo which is local) and run yumdownloader and it downloads
the one that is the highest version of anaconda-runtime.

yumdownloader -c odd.conf anaconda-runtime
anaconda-runtime-11.4.0.82-4.i386.rpm                      | 586 kB     00:03  


You're not running any yum plugins are you?
Comment 11 Matt Hicks 2008-07-17 13:28:35 EDT
Still hitting the problem.  I've tried to simplify the configuration a bit.

Here are all the yum RPMs I have installed:
[mhicks@localhost conf.d]$ rpm -qa | grep yum-
yum-metadata-parser-1.1.2-8.fc9.i386
yum-utils-1.1.14-4.fc9.noarch
yum-packagekit-0.1.12-13.20080522.fc9.i386
yum-3.2.16-2.fc9.noarch

Here is the simplified configuration (sorry, still Red Hat network only).  I
saved it in my home directory as test.repo:
########################################

[main]
keepcache=0
gpgcheck=0
plugins=0
reposdir=
tsflags=nodocs

[anaconda-extrarepo-1]
name=anaconda extra repo 1
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/
enabled=1

[anaconda-extrarepo-2]
name=anaconda extra repo 2
baseurl=http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-i386
enabled=1

[anaconda-extrarepo-3]
name=anaconda extra repo 3
baseurl=http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-noarch
enabled=1

[anaconda-extrarepo-4]
name=anaconda extra repo 4
baseurl=http://www.kanarip.com/anaconda/f9/i386/
enabled=1

[anaconda-extrarepo-5]
name=anaconda extra repo 5
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/updates/9/i386/
enabled=1

###############################################

Here is the command I ran:
[mhicks@localhost conf.d]$ sudo yumdownloader -c ~/test.repo anaconda-runtime

Output:
http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-i386/anaconda-runtime-11.4.0.82-4.i386.rpm:
[Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Traceback (most recent call last):
  File "/usr/bin/yumdownloader", line 277, in <module>
    util = YumDownloader()
  File "/usr/bin/yumdownloader", line 42, in __init__
    self.main()
  File "/usr/bin/yumdownloader", line 80, in main
    self.downloadPackages(opts)
  File "/usr/bin/yumdownloader", line 214, in downloadPackages
    path = repo.getPackage(download)
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 678, in getPackage
    cache=cache
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 661, in _getFile
    raise Errors.NoMoreMirrorsRepoError, errstr
yum.Errors.NoMoreMirrorsRepoError: failure:
anaconda-runtime-11.4.0.82-4.i386.rpm from anaconda-extrarepo-2: [Errno 256] No
more mirrors to try.

Comment 12 seth vidal 2008-07-17 13:40:31 EDT
okay, but this is correct:
it's getting the correct version -4 is the highest version in any repo

Try something for me - b/c this might be a simpler bug.

for repo-2 and repo-3 - make sure the baseurl's all have trailing '/''s

Comment 13 Matt Hicks 2008-07-17 13:50:54 EDT
Same results:

*************  COMMAND ******************
[mhicks@localhost ~]$ rpm -qa | grep yum-
yum-metadata-parser-1.1.2-8.fc9.i386
yum-utils-1.1.14-4.fc9.noarch
yum-packagekit-0.1.12-13.20080522.fc9.i386
yum-3.2.16-2.fc9.noarch

*************  COMMAND ******************
[mhicks@localhost ~]$ cat ~/test.repo 
[main]
keepcache=0
gpgcheck=0
plugins=0
reposdir=
tsflags=nodocs

[anaconda-extrarepo-1]
name=anaconda extra repo 1
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Everything/i386/os/
enabled=1

[anaconda-extrarepo-2]
name=anaconda extra repo 2
baseurl=http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-i386/
enabled=1

[anaconda-extrarepo-3]
name=anaconda extra repo 3
baseurl=http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-noarch/
enabled=1

[anaconda-extrarepo-4]
name=anaconda extra repo 4
baseurl=http://www.kanarip.com/anaconda/f9/i386/
enabled=1

[anaconda-extrarepo-5]
name=anaconda extra repo 5
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/updates/9/i386/
enabled=1

**************  COMMAND ******************
[mhicks@localhost ~]$ sudo yumdownloader -c ~/test.repo anaconda-runtime
http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-i386/anaconda-runtime-11.4.0.82-4.i386.rpm:
[Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Traceback (most recent call last):
  File "/usr/bin/yumdownloader", line 277, in <module>
    util = YumDownloader()
  File "/usr/bin/yumdownloader", line 42, in __init__
    self.main()
  File "/usr/bin/yumdownloader", line 80, in main
    self.downloadPackages(opts)
  File "/usr/bin/yumdownloader", line 214, in downloadPackages
    path = repo.getPackage(download)
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 678, in getPackage
    cache=cache
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 661, in _getFile
    raise Errors.NoMoreMirrorsRepoError, errstr
yum.Errors.NoMoreMirrorsRepoError: failure:
anaconda-runtime-11.4.0.82-4.i386.rpm from anaconda-extrarepo-2: [Errno 256] No
more mirrors to try.

Comment 14 seth vidal 2008-07-17 13:57:48 EDT
okay, just to be sure I'm not losing my mind. Could you update to yum 3.2.17
from updates-testing and test this command one more time?
Comment 15 Matt Hicks 2008-07-17 14:10:28 EDT
Still getting the same results after the update.  Is there something with
regards to my system state that I could be missing?

********** COMMAND **************
[mhicks@localhost ~]$ rpm -qa | grep yum
yum-metadata-parser-1.1.2-8.fc9.i386
yum-utils-1.1.14-4.fc9.noarch
yum-packagekit-0.1.12-13.20080522.fc9.i386
yum-3.2.16-2.fc9.noarch


********** COMMAND **************
[mhicks@localhost ~]$ sudo yum install --enablerepo=updates-testing yum
Loaded plugins: refresh-packagekit
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package yum.noarch 0:3.2.17-2.fc9 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size 
=============================================================================
Updating:
 yum                     noarch     3.2.17-2.fc9     updates-testing   718 k

Transaction Summary
=============================================================================
Install      0 Package(s)         
Update       1 Package(s)         
Remove       0 Package(s)         

Total download size: 718 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): yum-3.2.17-2.fc9.noarch.rpm                                             
                                                          | 718 kB     00:00     
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 30c9ecf8
Importing GPG key 0x30C9ECF8 "Fedora Project (Test Software)
<rawhide@redhat.com>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-test
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : yum                                               [1/2] 
  Cleanup        : yum                                               [2/2] 

Updated: yum.noarch 0:3.2.17-2.fc9
Complete!


********** COMMAND **************
[mhicks@localhost ~]$ rpm -qa | grep yum
yum-metadata-parser-1.1.2-8.fc9.i386
yum-utils-1.1.14-4.fc9.noarch
yum-packagekit-0.1.12-13.20080522.fc9.i386
yum-3.2.17-2.fc9.noarch


********** COMMAND **************
[mhicks@localhost ~]$ sudo yumdownloader -c ~/test.repo anaconda-runtime
http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-i386/anaconda-runtime-11.4.0.82-4.i386.rpm:
[Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Traceback (most recent call last):
  File "/usr/bin/yumdownloader", line 277, in <module>
    util = YumDownloader()
  File "/usr/bin/yumdownloader", line 42, in __init__
    self.main()
  File "/usr/bin/yumdownloader", line 80, in main
    self.downloadPackages(opts)
  File "/usr/bin/yumdownloader", line 214, in downloadPackages
    path = repo.getPackage(download)
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 676, in getPackage
    cache=cache
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 659, in _getFile
    raise Errors.NoMoreMirrorsRepoError, errstr
yum.Errors.NoMoreMirrorsRepoError: failure:
anaconda-runtime-11.4.0.82-4.i386.rpm from anaconda-extrarepo-2: [Errno 256] No
more mirrors to try.
Comment 16 seth vidal 2008-07-17 16:41:08 EDT
Though it should make no difference at all. please clean out all your yum caches.
Comment 17 Matt Hicks 2008-07-17 17:07:50 EDT
Same results...

*************** COMMAND *********************
[mhicks@localhost Desktop]$ rpm -qa | grep yum
yum-metadata-parser-1.1.2-8.fc9.i386
yum-utils-1.1.14-4.fc9.noarch
yum-packagekit-0.1.12-13.20080522.fc9.i386
yum-3.2.17-2.fc9.noarch


*************** COMMAND *********************
[mhicks@localhost Desktop]$ sudo yum clean all
Loaded plugins: refresh-packagekit
Cleaning up Everything


*************** COMMAND *********************
[mhicks@localhost Desktop]$ sudo yumdownloader -c ~/test.repo anaconda-runtime
anaconda-extrarepo-1                                     | 2.4 kB     00:00     
anaconda-extrarepo-3                                     |  951 B     00:00     
primary.xml.gz                                           | 7.8 kB     00:00     
anaconda-extrarepo-3                                           31/31
anaconda-extrarepo-2                                     |  951 B     00:00     
Not using downloaded repomd.xml because it is older than what we have
anaconda-extrarepo-5                                     | 2.3 kB     00:00     
anaconda-extrarepo-4                                     |  951 B     00:00     
http://genome-external.usersys.redhat.com/cobbler/repo_mirror/Fedora-9-genome-i386/anaconda-runtime-11.4.0.82-4.i386.rpm:
[Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Traceback (most recent call last):
  File "/usr/bin/yumdownloader", line 277, in <module>
    util = YumDownloader()
  File "/usr/bin/yumdownloader", line 42, in __init__
    self.main()
  File "/usr/bin/yumdownloader", line 80, in main
    self.downloadPackages(opts)
  File "/usr/bin/yumdownloader", line 214, in downloadPackages
    path = repo.getPackage(download)
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 676, in getPackage
    cache=cache
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 659, in _getFile
    raise Errors.NoMoreMirrorsRepoError, errstr
yum.Errors.NoMoreMirrorsRepoError: failure:
anaconda-runtime-11.4.0.82-4.i386.rpm from anaconda-extrarepo-2: [Errno 256] No
more mirrors to try.
Comment 18 seth vidal 2008-07-21 09:03:30 EDT
So, we've got the right version just somehow we're getting the repo confused.
Comment 19 Jan Hutař 2008-08-07 08:33:27 EDT
Hello,
does the repo have correct metadata? I mean, what does this say?

# yumdownloader -c ~/test.repo anaconda-runtime --disablerepo=\* --enablerepo=anaconda-extrarepo-1   # this repo do not contain the package, right?
# yumdownloader -c ~/test.repo anaconda-runtime --disablerepo=\* --enablerepo=anaconda-extrarepo-3   # this repo do contain the package, right?
Comment 20 Matt Hicks 2008-08-07 11:49:29 EDT
I think this must have been a yum repo problem.  Going through my standard test path now works:

**************************************************************
[root@localhost ~]# rpm -qa | grep yum
yum-metadata-parser-1.1.2-8.fc9.i386
yum-3.2.17-2.fc9.noarch
yum-packagekit-0.2.3-6.fc9.i386
yum-utils-1.1.14-4.fc9.noarch

[root@localhost ~]# sudo yum clean all
Loaded plugins: refresh-packagekit
Cleaning up Everything

[root@localhost ~]# yum clean all
Loaded plugins: refresh-packagekit
Cleaning up Everything

[root@localhost ~]# yumdownloader -c ~/test.repo anaconda-runtime
anaconda-runtime-11.4.0.82-4.i386.rpm                                                      | 586 kB     00:04     
**************************************************************

I don't own the end repositories so I don't know if any updates have been performed.  Thanks for the help and closing the bug.

Note You need to log in before you can comment on or make changes to this bug.