Bug 1019103 - librepo ignores mm0:alternates
librepo ignores mm0:alternates
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: librepo (Show other bugs)
20
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Mlcoch
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-15 03:38 EDT by Ales Kozumplik
Modified: 2014-09-30 19:41 EDT (History)
4 users (show)

See Also:
Fixed In Version: librepo-1.4.0-1.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-24 18:45:44 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
core hanging in select() (13.91 MB, application/x-core)
2013-10-15 03:38 EDT, Ales Kozumplik
no flags Details
Traceback (530 bytes, text/plain)
2013-11-20 03:22 EST, lnie
no flags Details

  None (edit)
Description Ales Kozumplik 2013-10-15 03:38:53 EDT
Created attachment 812376 [details]
core hanging in select()

I probably have a problem with network this morning and can not connect to the fedora updates mirror. I found out that librepo will never timeout in this particular case:

#! /usr/bin/python

import librepo

URL = 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f19&arch=x86_64'

def main():
    h = librepo.Handle()
    h.setopt(librepo.LRO_MIRRORLIST, URL)
    h.repotype = librepo.LR_YUMREPO
    h.destdir = './'
    r = librepo.Result()
    h.interruptible = True
    h.connecttimeout = 1
    h.perform(r)

if __name__ == '__main__':
    main()

/////


Outputs when run and ^C-ed after cca 15s:

$ time ./main.py 
^CTraceback (most recent call last):
  File "./main.py", line 18, in <module>
    main()
  File "./main.py", line 15, in main
    h.perform(r)
  File "/home/akozumpl/repos/librepo/build/librepo/python/python2/librepo/__init__.py", line 983, in perform
    _librepo.Handle.perform(self, result)
KeyboardInterrupt

real	0m14.947s
user	0m0.137s
sys	0m0.175s

Also attaching core dump after sending another process the ABRT signal (also after some 10s).
Comment 1 Tomas Mlcoch 2013-10-15 05:01:56 EDT
Hi,
I was able to reproduce this, but it's not a bug (or at least not on the librepo side), librepo doesn't hang, it is just trying other mirrors. For the metalink you've used it means 184 mirrors, and it takes some time.

It is all because the metalink and the mirrors are out of sync.

This is the current metalink: https://paste.fedoraproject.org/46787/
This is the current repomd.xml from the first mirror: https://paste.fedoraproject.org/46788/ (ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/updates/19/x86_64/repodata/repomd.xml)

sha256sum of the repomd.xml is 90afabba594d0e1c9fa9656e6c174e8adebf099a73d39e82c9c54682f65ab6ac

But the expected one (loaded from metalink) is d4f9ad66f7c6e000d8ebf9ec92ad2c4636547853708554d93dab672bdfd98ca1

I suggest use of h.maxmirrortries option.
Comment 2 Tomas Mlcoch 2013-10-15 06:05:02 EDT
Or you can turn off checksum check by the

h.checksum = False

From dnf point of view, librebo should provide a better mechanism to inform about what is wrong after each mirror failure during download of metadata.
Something like LrMirrorFailureCb which is used during download of packages.
I will talk with zpavlas about this.
Comment 3 Ales Kozumplik 2013-10-15 07:32:59 EDT
(In reply to Tomas Mlcoch from comment #2)
> From dnf point of view, librebo should provide a better mechanism to inform
> about what is wrong after each mirror failure during download of metadata.
> Something like LrMirrorFailureCb which is used during download of packages.
> I will talk with zpavlas about this.

Thanks!
Comment 4 Ales Kozumplik 2013-10-15 07:35:17 EDT
(In reply to Tomas Mlcoch from comment #1)
> Hi,
> I was able to reproduce this, but it's not a bug (or at least not on the
> librepo side), librepo doesn't hang, it is just trying other mirrors. For
> the metalink you've used it means 184 mirrors, and it takes some time.

Thanks for clarifying, what would be the recommended value to the h.maxmirrortries option? I want a safe value which however won't let the user wait 184*timeout seconds (if I count it right) for a failure.

Also, we should probably report this to the fedora infrastructure because it is clearly not dnf/librepo's fault.
Comment 5 Patrick Uiterwijk 2013-10-15 07:39:57 EDT
If the metalink info and mirror content's checksum don't match, that's an error in the Fedora Infra.

I will look into it.
Comment 6 Patrick Uiterwijk 2013-10-15 07:48:27 EDT
When I run sha256sum over both the link you give (ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/updates/19/x86_64/repodata/repomd.xml) or another random mirror (ftp://mirror.1000mbps.com/fedora/linux/updates/19/x86_64/repodata/repomd.xml) I get the value:
099ff51a491eba3f16646efb5d16319249e0bb10f1dc1476d47e23b25e0e0f94

Which IS a valid checksum according to the metalink (check <mm0:alternatives>).

Could it be that you have a caching proxy that returns a stale repomd, or that the mirror you used is stale?
Comment 7 Patrick Uiterwijk 2013-10-15 08:31:28 EDT
Okay, the summary is that librepo ignores mm0:alternates.

mm0:alternates was introduced because there's some delay between a new repomd file and the mirrors updating: we add mm:alternatives for some previous repomd revisions, so that one or two previous revisions can also be verified as correct until that mirror is synced.
Comment 8 Tomas Mlcoch 2013-10-16 07:40:00 EDT
Patric,
Thank you for clarifying!

Fixed in head: https://github.com/Tojaj/librepo/commit/04ac52b072f3a23cf7b3177c96a23e6665e81f25
Comment 9 Fedora Update System 2013-11-19 03:52:12 EST
librepo-1.4.0-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/librepo-1.4.0-1.fc20
Comment 10 Fedora Update System 2013-11-19 16:50:33 EST
Package librepo-1.4.0-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing librepo-1.4.0-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-21715/librepo-1.4.0-1.fc20
then log in and leave karma (feedback).
Comment 11 lnie 2013-11-20 03:21:14 EST
Tested with librepo-1.4.0-1.fc20 (x86_64),seems fine,the output has been attached.
Comment 12 lnie 2013-11-20 03:22:21 EST
Created attachment 826466 [details]
Traceback
Comment 13 Tomas Mlcoch 2013-11-20 03:36:59 EST
Hi Inie,
as I could see in the traceback, it seems there was a connection problem
Are you sure, the link was working at the time when it was tested?
Comment 14 lnie 2013-11-20 03:43:58 EST
Hi Tomas,

 I turned off the network intentionally.From #comment0 :-),I think it's nessary to reproduce it
Comment 15 lnie 2013-11-20 03:46:42 EST
>I probably have a problem with network this morning and can not connect to the >fedora updates mirror. I found out that librepo will never timeout in this >particular case:
Comment 16 Fedora Update System 2013-11-24 18:45:44 EST
librepo-1.4.0-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.