Created attachment 995110 [details] anaconda log Description of problem: We are trying to automate anaconda testing and from time to time (approx. one in ten runs) installation from netinst fails during downloading packages with 500 or 404 HTTP code. Interesting is that it happens on different mirrors and when I check it by hand, given mirror actually contains request package. I was able to reproduce it on both, virtual machine and bare metal. It happens on mirrors here in Brno (vutbr.cz and fi.muni.cz) and in Boston. I am attaching logs from one failed installation, but I can provide other logs from other runs (it happens fairly often). Version-Release number of selected component (if applicable): 22.20.1-1
Created attachment 995111 [details] dnf log
Created attachment 995112 [details] hawkey log
Created attachment 995113 [details] ifcfg
Created attachment 995114 [details] packaging log
Created attachment 995115 [details] program log
Created attachment 995116 [details] sensitive-info log
Created attachment 995117 [details] storage log
Created attachment 995118 [details] syslog
Created attachment 995119 [details] packaging log from different machine (in Boston)
Created attachment 995125 [details] another packaging log
Although it might seem like it's problem with mirrors, it happens so often and on wide range of mirrors so it seems it could be problem with software used for downloading. It didn't happened with F21, maybe yum retried failed downloads and dnf does not?
dnf handles package downloads, it looks like it is moving to the next mirror, but for whatever reason the mirror and metadata are out of sync. Maybe dnf has some thoughts on it.
It is weird that anaconda says that it got 404 when requesting package, but when you try it by hand, that mirror contains that package. I have tried it and immediately after anaconda failed with 404 during downloading package, I ran curl on that link and it worked without problem. Another weird thing is, when you look on this log: https://bugzilla.redhat.com/attachment.cgi?id=995125 anaconda gets FTP 550 when requesting package (I have tried it back then and I was able to download that package) and then immediately it tries the same mirror, only via HTTP and it gets HTTP 404 (although again, I was able to download that package via curl).
Thanks for the report. Can you try that again with dnf-0.6.4 and librepo-1.7.13 setting `timeout` config option [1] to higher number (optionally setting metadata_expire to low value) and share the result, please? [1] http://dnf.readthedocs.org/en/latest/conf_ref.html?highlight=timeout#options-for-both-main-and-repo
Thanks for info, Jan Silhan. Brian, can we manually play with these options inside the anaconda environment, and how? We can create an updates.img if needed, just tell us what to edit (config files in /etc or anaconda source files). Thanks. Btw, another option that caught my attention is "minrate", and I'm not sure why it is '0' by default and whether it has some special meaning. But it could help us also with an issue related to bug 1196629, when downloading metadata in the main hub takes 5-10 minutes (which makes our automated tests fail, and is quite user unfriendly), and trying a different mirror automatically could solve it all.
You can adjust those in the _configure method, here: https://github.com/rhinstaller/anaconda/blob/f22-branch/pyanaconda/packaging/dnfpayload.py#L348
You can play around with `minrate` but by default the remaining seconds to timeout is count when 0 bytes are transferred - IMO good enough.
Unfortunately, I am now unable to reproduce this, so I am unable to play with `timeout` and `minrate` options. Even machines that are running our tests hadn't encountered this bug since 24. 2., but they have encountered it several times before that.
Closing this. The new version of librepo improves the mirror selection mechanism so that could be the reason why you can't reproduce it. Either way I am closing this, feel free to reopen once you find the reproducer.