Description of problem: dnf fails to download packages from ftp sites. Terminal messages show that dnf is busy retrying broken links and performing checksum comparisons. The result is a lot of data being downloaded and discarded. Eventually dnf gives up on the task and exits. One can manually pull the broken packages from a web-browser by substituting the "ftp://" prefix with "http://" (an indication that the resource exists). Running the same task with yum further confirms an anomaly. Yum finishes the depsolving slower but is able to finish the session. Yum skips URLs that are broken after first trial and doesn't choke on checksum comparisons. Something is definitely wrong somewhere. Version-Release number of selected component (if applicable): dnf-0.5.5-1.fc22.noarch How reproducible: Have persisted for several weeks Steps to Reproduce: 1. run an update with dnf (try with a mobile broadband) 2. watch the terminal outputs 3. Actual results: http://fpaste.org/126000/ the retrials and checks are expensive. Sometimes truncated versions of packages larger than 10MB are downloaded several times and discarded. Expected results: smooth URL resolutions (especially ftp links), skip unreliable servers (no expensive retrials). Is there no way to perform checksum comparisons before actual downloads? Additional info:
Including fpaste content here (since fpaste expires over time). MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Curl error: Remote file not found for ftp://be.mirror.eurid.eu/fedora/linux/development/rawhide/x86_64/os/drpms/gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Curl error: Requested range was not delivered by the server for http://mirror.clarkson.edu/fedora/linux/development/rawhide/x86_64/os/drpms/gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm [MIRROR] gdb-7.8-18.fc22.x86_64.rpm: Curl error: Remote file not found for ftp://be.mirror.eurid.eu/fedora/linux/development/rawhide/x86_64/os/Packages/g/gdb-7.8-18.fc22.x86_64.rpm [MIRROR] gdb-7.8-18.fc22.x86_64.rpm: Curl error: Requested range was not delivered by the server for http://mirror.clarkson.edu/fedora/linux/development/rawhide/x86_64/os/Packages/g/gdb-7.8-18.fc22.x86_64.rpm [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Curl error: Remote file not found for ftp://be.mirror.eurid.eu/fedora/linux/development/rawhide/x86_64/os/drpms/dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Curl error: Requested range was not delivered by the server for http://mirror.clarkson.edu/fedora/linux/development/rawhide/x86_64/os/drpms/dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gdb-7.8-18.fc22.x86_64.rpm: Curl error: Transferred a partial file for http://mirror.cogentco.com/pub/linux/fedora/linux/development/rawhide/x86_64/os/Packages/g/gdb-7.8-18.fc22.x86_64.rpm [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] gdb-7.8-18.fc22.x86_64.rpm: Downloading successfull, but checksum doesn't match. Expected: 5dbe4f03868c87e8ee013fd1961cce9af81e62140d641b77036970e96622d6d9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Curl error: Couldn't connect to server for http://fedora.uib.no/fedora/linux/development/rawhide/x86_64/os/drpms/gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Curl error: Remote file not found for ftp://ftp.kaist.ac.kr/fedora/development/rawhide/x86_64/os/drpms/gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm [MIRROR] gedit-3.13.4-1.fc22_3.13.5-1.fc22.x86_64.drpm: Downloading successfull, but checksum doesn't match. Expected: 65dc0eb45c15386909a59b338c8125fbeb1e22febba74ac9f1fad14d1d85bc1f(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256) [MIRROR] dnf-0.5.5-1.fc22_0.6.0-1.fc22.noarch.drpm: Downloading successfull, but checksum doesn't match. Expected: ac1656e431ef79aac736ddcefafdbc7aee19271e1355111b146f78a5e35c7ce9(sha256)
I see this as well, even on university connection (high speed, high reliability). As I recall, I didn't use to see this several months ago, also, sometimes running DNF even fails to download the packages at all, while retrying (sometimes several times) immediately after fail makes the issue almost magically disappear, or at least not to fail completely... It also seems to be getting worse recently.
Oh, looking at the bug data: I forgot to mention, I'm seeing this on F20, dnf-0.5.4-2.fc20.noarch.
Hi, thanks for the report. Tomas, can you take a look, please? Feel free to change component back to dnf if you consider it as dnf issue.
Another info: running dnf clean all before dnf upgrade/install seems to fix the issue, i.e. even when dnf upgrade spams the console with [MIRROR] error messages (repeatedly, even after running the same command again and again and again...), issuing the same command after dnf clean all makes the errors disappear, completely. I guess there might be some cache/metadata issue behind this behaviour. Hope this helps.
hmmm, can someone upload somewhere metadata from `/var/cache/dnf` with wrong checksums and then the correct ones after executing `dnf clean all`, please?
Hi Jan, What files do you need? I have had a similar but very different issue with dnf / librepo. Instead of dnf "wasting all the bandwidth", I've had it try at a mirror once, then give up entirely. The same process through yum completes successfully. [root@KnightRider ~]# dnf update --enablerepo="*testing" RPM Fusion for Fedora 20 - Nonfree - Test Updates 9.7 kB/s | 2.6 kB 00:00 Fedora 20 - x86_64 - Test Updates 1.0 MB/s | 3.2 MB 00:03 RPM Fusion for Fedora 20 - Free - Test Updates 15 kB/s | 2.0 kB 00:00 Dependencies resolved. ============================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================== Installing: baloo-akonadi x86_64 4.13.3-3.fc20 updates-testing 74 k replacing baloo.x86_64 4.13.3-1.fc20 python-ceph x86_64 1:0.80.5-6.fc20 updates-testing 78 k ceph-common x86_64 1:0.80.5-6.fc20 updates-testing 4.9 M python-flask noarch 1:0.10.1-3.fc20 fedora 204 k python-itsdangerous noarch 0.23-1.fc20 fedora 24 k python-werkzeug noarch 0.9.4-1.fc20 fedora 565 k ceph-libs-compat x86_64 1:0.80.5-6.fc20 updates-testing 12 k replacing ceph-libs.x86_64 0.81.0-5.fc20 librados2 x86_64 1:0.80.5-6.fc20 updates-testing 1.5 M replacing ceph-libs.x86_64 0.81.0-5.fc20 librbd1 x86_64 1:0.80.5-6.fc20 updates-testing 342 k replacing ceph-libs.x86_64 0.81.0-5.fc20 qt-x11 i686 1:4.8.6-10.fc20 updates 13 M libcephfs1 x86_64 1:0.80.5-6.fc20 updates-testing 1.6 M replacing ceph-libs.x86_64 0.81.0-5.fc20 Upgrading: akregator x86_64 4.13.3-12.fc20 updates-testing 1.4 M akregator-libs x86_64 4.13.3-12.fc20 updates-testing 144 k kdepim-common x86_64 4.13.3-12.fc20 updates-testing 744 k kdepim-libs x86_64 7:4.13.3-12.fc20 updates-testing 2.1 M kdepim x86_64 7:4.13.3-12.fc20 updates-testing 5.9 k kalarm x86_64 4.13.3-12.fc20 updates-testing 1.0 M kontact x86_64 4.13.3-12.fc20 updates-testing 922 k knotes x86_64 4.13.3-12.fc20 updates-testing 258 k knode x86_64 4.13.3-12.fc20 updates-testing 1.2 M blogilo x86_64 4.13.3-12.fc20 updates-testing 487 k ktimetracker x86_64 4.13.3-12.fc20 updates-testing 408 k kjots x86_64 4.13.3-12.fc20 updates-testing 449 k korganizer x86_64 4.13.3-12.fc20 updates-testing 1.1 M kmail x86_64 4.13.3-12.fc20 updates-testing 2.2 M kleopatra x86_64 4.13.3-12.fc20 updates-testing 1.4 M kaddressbook x86_64 4.13.3-12.fc20 updates-testing 200 k kontact-libs x86_64 4.13.3-12.fc20 updates-testing 63 k knotes-libs x86_64 4.13.3-12.fc20 updates-testing 130 k knode-libs x86_64 4.13.3-12.fc20 updates-testing 613 k korganizer-libs x86_64 4.13.3-12.fc20 updates-testing 366 k kmail-libs x86_64 4.13.3-12.fc20 updates-testing 2.7 M kleopatra-libs x86_64 4.13.3-12.fc20 updates-testing 468 k kaddressbook-libs x86_64 4.13.3-12.fc20 updates-testing 249 k baloo-libs x86_64 4.13.3-3.fc20 updates-testing 203 k baloo x86_64 4.13.3-3.fc20 updates-testing 47 k baloo-file x86_64 4.13.3-3.fc20 updates-testing 190 k ca-certificates noarch 2014.2.1-1.0.fc20 updates-testing 406 k ceph x86_64 1:0.80.5-6.fc20 updates-testing 9.3 M chrony x86_64 1.30-2.fc20 updates-testing 262 k cinnamon-screensaver x86_64 2.2.4-3.fc20 updates-testing 98 k devscripts x86_64 2.14.6-2.fc20 updates-testing 429 k devscripts-minimal x86_64 2.14.6-2.fc20 updates-testing 31 k dhclient x86_64 12:4.2.7-2.fc20 updates-testing 280 k dhcp-common x86_64 12:4.2.7-2.fc20 updates-testing 173 k dhcp-libs x86_64 12:4.2.7-2.fc20 updates-testing 126 k dhcp x86_64 12:4.2.7-2.fc20 updates-testing 516 k firewall-config noarch 0.3.11-1.fc20 updates-testing 104 k firewalld noarch 0.3.11-1.fc20 updates-testing 462 k ghostscript x86_64 9.14-4.fc20 updates-testing 4.4 M ghostscript-devel x86_64 9.14-4.fc20 updates-testing 50 k gparted x86_64 0.18.0-2.fc20 updates-testing 1.7 M jakarta-commons-httpclient noarch 1:3.1-15.fc20 updates-testing 241 k kexec-tools x86_64 2.0.4-28.fc20 updates-testing 317 k libkgeomap x86_64 4.2.0-5.fc20 updates-testing 228 k libnfsidmap x86_64 0.26-0.0.fc20 updates-testing 46 k libreport x86_64 2.2.3-2.fc20 updates-testing 405 k libreport-python x86_64 2.2.3-2.fc20 updates-testing 63 k libreport-filesystem x86_64 2.2.3-2.fc20 updates-testing 35 k libreport-anaconda x86_64 2.2.3-2.fc20 updates-testing 43 k libreport-cli x86_64 2.2.3-2.fc20 updates-testing 47 k libreport-fedora x86_64 2.2.3-2.fc20 updates-testing 40 k libreport-gtk x86_64 2.2.3-2.fc20 updates-testing 94 k libreport-newt x86_64 2.2.3-2.fc20 updates-testing 43 k libreport-plugin-bugzilla x86_64 2.2.3-2.fc20 updates-testing 79 k libreport-plugin-kerneloops x86_64 2.2.3-2.fc20 updates-testing 45 k libreport-plugin-logger x86_64 2.2.3-2.fc20 updates-testing 48 k libreport-plugin-reportuploader x86_64 2.2.3-2.fc20 updates-testing 52 k libreport-plugin-ureport x86_64 2.2.3-2.fc20 updates-testing 52 k libreport-python3 x86_64 2.2.3-2.fc20 updates-testing 49 k libreport-web x86_64 2.2.3-2.fc20 updates-testing 46 k libserf x86_64 1.3.7-1.fc20 updates-testing 53 k libteam x86_64 1.12-1.fc20 updates-testing 46 k teamd x86_64 1.12-1.fc20 updates-testing 108 k libtirpc x86_64 0.2.5-0.0.fc20 updates-testing 85 k libwacom x86_64 0.10-1.fc20 updates-testing 27 k libwacom-data noarch 0.10-1.fc20 updates-testing 39 k numpy x86_64 1:1.8.2-1.fc20 updates-testing 2.9 M openal-soft x86_64 1.16.0-1.fc20 updates-testing 319 k openal-soft i686 1.16.0-1.fc20 updates-testing 320 k openal-soft-devel x86_64 1.16.0-1.fc20 updates-testing 45 k perl-Socket x86_64 1:2.015-1.fc20 updates-testing 50 k perl-XML-TreeBuilder noarch 5.4-0.fc20 updates-testing 21 k php-pear noarch 1:1.9.5-2.fc20 updates-testing 360 k polkit-qt x86_64 0.112.0-1.fc20 updates-testing 72 k poppler-data noarch 0.4.7-1.fc20 updates-testing 2.2 M ppp x86_64 2.4.5-34.fc20 updates-testing 359 k python-pillow x86_64 2.2.1-5.fc20 updates-testing 455 k rubygem-nokogiri x86_64 1.6.3.1-1.fc20.1 updates-testing 523 k satyr x86_64 0.14-2.fc20 updates-testing 485 k sqlite x86_64 3.8.6-1.fc20 updates-testing 433 k sqlite i686 3.8.6-1.fc20 updates-testing 440 k sqlite-devel x86_64 3.8.6-1.fc20 updates-testing 109 k subversion x86_64 1.8.10-1.fc20 updates-testing 1.2 M subversion-libs x86_64 1.8.10-1.fc20 updates-testing 1.1 M subversion-perl x86_64 1.8.10-1.fc20 updates-testing 868 k subversion-javahl x86_64 1.8.10-1.fc20 updates-testing 374 k subversion-python x86_64 1.8.10-1.fc20 updates-testing 697 k vim-common x86_64 2:7.4.402-1.fc20 updates-testing 5.9 M vim-enhanced x86_64 2:7.4.402-1.fc20 updates-testing 1.0 M vim-filesystem x86_64 2:7.4.402-1.fc20 updates-testing 11 k vim-minimal x86_64 2:7.4.402-1.fc20 updates-testing 439 k xen-libs x86_64 4.3.2-7.fc20 updates-testing 427 k xen-licenses x86_64 4.3.2-7.fc20 updates-testing 84 k httpcomponents-client noarch 4.2.5-4.fc20 updates-testing 425 k hwdata noarch 0.269-1.fc20 updates-testing 1.3 M Transaction Summary ============================================================================================================================================================================================================================================== Install 11 Packages Upgrade 95 Packages Total download size: 84 M Is this ok [y/N]: yy Is this ok [y/N]: y Downloading Packages: (1/106): python-ceph-0.80.5-6.fc20.x86_64.rpm 112 kB/s | 78 kB 00:00 (2/106): baloo-akonadi-4.13.3-3.fc20.x86_64.rpm 105 kB/s | 74 kB 00:00 (3/106): ceph-common-0.80.5-6.fc20.x86_64.rpm 1.3 MB/s | 4.9 MB 00:03 (4/106): python-itsdangerous-0.23-1.fc20.noarch.rpm 4.7 kB/s | 24 kB 00:05 (5/106): python-flask-0.10.1-3.fc20.noarch.rpm 40 kB/s | 204 kB 00:05 (6/106): ceph-libs-compat-0.80.5-6.fc20.x86_64.rpm 31 kB/s | 12 kB 00:00 (7/106): python-werkzeug-0.9.4-1.fc20.noarch.rpm 251 kB/s | 565 kB 00:02 (8/106): librbd1-0.80.5-6.fc20.x86_64.rpm 571 kB/s | 342 kB 00:00 [MIRROR] qt-x11-4.8.6-10.fc20.i686.rpm: Curl error: Failure when receiving data from the peer for ftp://mirror.nexicom.net/pub/fedora/linux/updates/20/x86_64/qt-x11-4.8.6-10.fc20.i686.rpm [FAILED] qt-x11-4.8.6-10.fc20.i686.rpm: Curl error: Failure when receiving data from the peer for ftp://mirror.nexicom.net/pub/fedora/linux/updates/20/x86_64/qt-x11-4.8.6-10.fc20.i686.rpm Error: Error downloading packages:c20.x86_64.rpm 22% [====================== ] 4.2 MB/s | 19 MB 00:15 ETA Curl error: Failure when receiving data from the peer for ftp://mirror.nexicom.net/pub/fedora/linux/updates/20/x86_64/qt-x11-4.8.6-10.fc20.i686.rpm
(In reply to Jan Silhan from comment #6) > hmmm, can someone upload somewhere metadata from `/var/cache/dnf` with wrong > checksums and then the correct ones after executing `dnf clean all`, please? I have archived metadata capturing the checksums for three consecutive times (each followed by 'dnf clean all'). The files are huge (totalling 335MB). I am on a restrictive bandwidth subsciption and attachment limit is 20MB. Please advise on how to make them available
Right, choose only one repo (the smallest) where downloads are failing and disable the rest. Archive dir /var/cache/dnf/<arch>/<fc>/<repo>/repodata/ and some downloaded package where checksum doesn't match. Then do that again after 'dnf clean all', please.
I will look at this (the needinfo still stands).
Martin, I think your particular case is that of bug 1121183. In this bug I will focus only on the original issue reported in comment 0.
Onyeibo, despite my best efforts (fedora rawhide, dnf-0.6.1, rawhide versions of curl and librepo, mobile internet connect) I am not able to reproduce, the packages are downloading fine. Can I ask you to try downloading some packages after cleaning up the cache, e.g. $ dnf clean all $ LIBREPO_DEBUG=True dnf download gdb* --enablerepo=rawhide &>out and report if the problem is reproducible that way and attach the 'out' file? Also, to be sure we're chasing a legit bug it would really help if bug 1121183 got fixed, setting a blocker. Thanks!
(I can confirm that fedora-repos-rawhide-22-0.1.noarch has bug 1121183 fixed.)
(de-priortizing, apparently not as urgent as originally seemed)
Is there still need for repodata samples? I'm hoping its not necessary any longer -- 50MB to 70MB of upload is still a significant bandwidth expense on a metered network.
It would help somewhat in debugging this. Let's start with the request from comment 12 please.
I have not had this issue after reinstalling and updating my rawhide machine for some days now. It would be chasing the wind to debug this under the current conditions. I don't get those checksum errors anymore. I also have a problem with the renaming of this request. The same network worked flawlessly with yum (in that period). I cannot imagine the criteria for categorizing the network as "slow/unstable". If you still want to chase this, I have uploaded the repodata snapshots here: http://bugs.onyeibo.fastmail.fm.user.fm/errors-rawhide-repodata.tar.xz (that's with checksum errors) http://bugs.onyeibo.fastmail.fm.user.fm/smooth-rawhide-repodata.tar.xz (that's running without checksum errors) Goodluck.
OK, thank you for the reply. It could even be that one of the components *below* librepo was broken as rawhide rolls on rather wildly. Closing this as unreproducible, if the same problem appears again please supply the librepo debug output as described in comment 12. Compared to other download/md issues this bug should still be considered special due to the fact that the problem persisted after cleaning all md as described in comment 8. And I changed the summary based on comment 1 statement that this is reproducible on mobile broadband.
I'm experiencing similar problem, suddenly today, whether with mirrors or without # dnf update gcc-4.9.2-2.fc22.x86_64.rpm: Downloading successfull, but checksum doesn't match. Expected: 50e845c5f3c0a46a5f47ac1f393126565ef0ff62094cc650c5942e4bcedc7638(sha256) plus a few more packages. I've clean cache, out.log attached
Created attachment 969097 [details] dnf update log
Reopening the bug for re-inspection.
when I do rpm -Va almost every file has - digest (formerly MD5 sum) differs. How is that even possible?? Another thing is on my second laptop - which is slightly different for it had upgrade from f20 to rawhide(f22) some months ago whereas this one had clean f22 installation - I don't see this problem. My concern is it might be worth letting kernel dev know, I get the feeling there might be something wrong with AMD new CPUs (HP EliteBook 745 G2 AMD A10 PRO-7350B R6, 10 Compute Cores 4C+6G).
also partitions are encrypted where the problem occurred, I'm guessing and trying to see the differences between two systems.
now, the weirdest bit, after rpm -Va I told gnome session to reboot, gnome asked back whether to install updates, I said yes, rebooted, and updates installation started before Xwindows/gdm and it all went in fine! So it appears. Will have to wait till new updates to see if it was one-off.
Hi lejeczek, firstly thanks for the report with log attached! This is very interesting because the error message says that expected checksum for "gcc-4.9.2-2.fc22.x86_64.rpm" was: 50e845c5f3c0a46a5f47ac1f393126565ef0ff62094cc650c5942e4bcedc7638 (sha256) But I downloaded the file from the mirror and I got: 6bfe4cfecf13ab4a83d8b4a8d56482d39811c297fbaecfbcef870b028fd1e53b (sha256) which is also a checksum that is expected by the current repo's metadata (primary.xml). Same situation for "bind-libs-lite-9.9.6-6.P1.fc22.x86_64.rpm" - From the log you've attached [1], the expected checksum was: 0ccc510c9213579116be043b59056d218f1652ee42089a9de6008d8eed2ae9c1 (sha256) while the real checksum is: 45a73569aa9c5a1b76f1d0283d7ff7ae8b03454a4e65966098d05763ba39bd79 (sha256) I have no idea how primary metadata could contain correct filenames (which are composed from name-version-release) and had bad checksums in the same time. Next time you see the error, could you please also grab a local cached copy of the metadata that was used by DNF and send it to me? For rawhide the path is: ls /var/cache/dnf/<ARCH>/<FEDORA_VERSION>/rawhide/repodata/*-primary.xml.gz E.g.: ls /var/cache/dnf/x86_64/21/rawhide/repodata/5d54d6c88e1d3de80daab17135e2273fdc6c3827efcf44e3156e0b0dbf6d9edd-primary.xml.gz Note: This file could be a quite big (e.g. 11Mb) so you probably won't be able to attach it to the bugzila, but it would be nice if you could share it with me by any other way, choice is up to you. Regards Tomas [1] https://bugzilla.redhat.com/attachment.cgi?id=969097
A side note: As could be seen in the log, the downloads of some file were restored (the exactly same filenames already existed in dnf cache and librepo just try to continue with these unfinished downloads. I.e. download only the missing pieces instead of downloading whole files again): select_next_target: Selecting mirror for: Packages/g/gcc-4.9.2-2.fc22.x86_64.rpm prepare_next_transfer: URL: http://download.fedoraproject.org/pub/fedora/linux/development/rawhide/x86_64/os/Packages/g/gcc-4.9.2-2.fc22.x86_64.rpm prepare_next_transfer: Used offset for download resume: 18544320 lr_download: Downloading started In that case (in case of resumed downloads), I could imagine that the files could get corrupted (somehow), but it doesn't explain why were expected different checksums - which is the problem here. -- Brain dump -- It's really rare and unusual that two packages with the same filenames have different checksums. Koji (Fedora's build system) doesn't let anyone to build a package with NVR (name-version-release) that already exists. Thus every package filename which is produced by Koji is unique. Of course, with exception for scratch builds - for scratch builds the uniqueness constrains don't apply, but scratch build are only for testing by developers and never goes to repositories. Another case when two packages with the exactly same filenames could have different checksums is in a case when they are signed with different keys (or one is signed and the second one is not). But AFAIK, unsigned packages should never get into any official fedora repo.
Created attachment 971924 [details] from /var/cache/dnf/x86_64/22/rawhide/repodata/
hit the problem again, today out.1.log dnf clean all out.2.log
Created attachment 971931 [details] log 1
Created attachment 971932 [details] log 2
Thanks! From what I see, the expected checksums are correct. The issue must be elsewhere. I see that a lot of downloads were resumed - i.e. there already existed a files with the same filenames but with smaller size that the expected ones. The main question is where did all this partially downloaded files come from? This could be a sign of some kind of race condition. Something like two or more processes downloading a same file into the shared dnf cache and thus writing simultaneously to a same file which would led to such errors. Could you please next time also append one of the partially downloaded files that were reported in a log as the one with non-matching checksum (the smallest one to save your bandwidth would be sufficient). The files should be stored in: /var/cache/dnf/<architecture>/<fedora_version>/fedora/packages/ E.g. /var/cache/dnf/x86_64/21/fedora/packages/ What is interesting, in my case when I manually break a dnf download then no files are kept in the cache, thus I wonder where all your partially downloaded files goes from.
Created attachment 972307 [details] download package
also, I do "clean all" thus I don't think any partial/previous/old downloads/cache play any part here, problem still exists.
also, yum update worked for packages dnf failed, not for all, I had to skip broken. Also dnf seems to miss some updates, yum update: yum update freerdp-libs fails for other dep problems, but dnf update freerdp-libs has/finds no updates even though: dnf info freerdp-libs | egrep '(^Ver|^Rel) Version : 1.2.0 Release : 0.2.beta.1.fc22 Version : 1.2.0 => installed Release : 0.4.beta.1.fc22 Version : 1.2.0 Release : 0.4.beta.1.fc22
*** Bug 1176105 has been marked as a duplicate of this bug. ***
(In reply to lejeczek from comment #33) > also, I do "clean all" thus I don't think any partial/previous/old > downloads/cache play any part here, problem still exists. I tried to manually delete the content of /var/cache/dnf/x86_64/ but interestingly enough, it still downloaded the DRPMs. Hence I added "deltarpm=false" to be able to update my system. This is just workaround though.
*** Bug 1177742 has been marked as a duplicate of this bug. ***
when I get this, I can fix it by manually removing the affected .rpm and .drpms in /var/cache/dnf/ It looks like if I've got a partial file dnf continues from the previous last download point. If, however, the partial file is corrupt, it always continues to append to a corrupt file, and thus is stuck in an infinite loop with bad checksums. I'll try and manually corrupt a large partial download.
above (38) does not do anything for me, besides - dnf clean - does this and should be no need for any manual intervention. I my case it's not a problem with partially downloaded / corrupted packages, nothing in local disk cache and still checksum doesn't match.
Thanks for collaboration to all. Issue with partially downloaded files from comment 38 (and maybe the others too) could be fixed when PR [1] solving bug 1157233 is merged. This change will be in dnf-0.6.4. [1] https://github.com/rpm-software-management/dnf/pull/182
there is definitely something wrong with dnf on my system. Today again (it happend before) my gnome notification told me there are some updates, asked to install them, "Restar & Install" and everything dnf a minute ago was failing about (checksums) went in perfectly fine.
(In reply to lejeczek from comment #41) Are you speaking about gnome-software updates? They use packagekit in background and not dnf.
exactly what I was saying - something wrong with dnf and it ain't partial/broken/incomplete package downloads.
This may be relevant: I don't use gnome-software updates because it downloads on expensive networks, and I've switched dnf metadata to very long for the same reason. I initiate manual downloads on cheap networks, dnf only. Therefore my pre-conditions are different to lejeczek's.
I've fiddled a bit with dnf.conf, added: keepcache=0 deltarpm=0 metadata_expire=3m this seems to solve my checksum mismatches. (for now at least, will do more fiddling)
*** Bug 1177691 has been marked as a duplicate of this bug. ***
*** Bug 1177833 has been marked as a duplicate of this bug. ***
*** Bug 1119977 has been marked as a duplicate of this bug. ***
Hi all, I've made several changes to Librepo that should fix this issue. These changes are now available in the librepo git and also new build for rawhide is ready and should be available in rawhide repository soon. The build: http://koji.fedoraproject.org/koji/taskinfo?taskID=8639651 In this version, librepo tries to resume any download only once. If this one attempt fails in next try the whole file is downloaded from scratch. Also, the librepo tries to resume only files that were originally being downloaded by librepo. This behavior is achieved by xattr flag, that is set for each downloaded file during downloading and is removed when the file is successfully downloaded. If a librepo process is interrupted during a download process, the partially downloaded files still have the flag and could be resumed. Other files are not considered trustworthy and such files will be downloaded from scratch.
librepo-1.7.12-1.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/librepo-1.7.12-1.fc21
Package librepo-1.7.12-1.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing librepo-1.7.12-1.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-0896/librepo-1.7.12-1.fc21 then log in and leave karma (feedback).
librepo-1.7.13-1.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/librepo-1.7.13-1.fc21
*** Bug 1185504 has been marked as a duplicate of this bug. ***
librepo-1.7.13-1.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
I've just tried the new librepo 1.7.13 and I did not see any improvement. I am on Fedora 21 x64 in a corporate environment behind a proxy. Fedora 21 is fully updated as of this morning 29/1/15 after "yum clean all", yum needs 5 minutes to download the metadata, trying various mirrors and doing partial downloads. The log is at the bottom of this comment. did the same with dnf and it tries forever to download a file. Fedora 21 - x86_64 37% [=======- ] 26 kB/s | 15 MB 16:08 ETA it goes almost at 100%, then starts again, then stops halfway, starts again.... After more that 1 hour nothing else happened. Other times, the target size is 39 MB, it varies. The proxy I need to use does some file inspection and sometimes blocks the download almost at the end while it checks the file and then continues. yum never had an issue with this, and after a few try / partial downloads, in 5 minutes finishes the job. here is the output of yum. I was not able to find a similar verbose output for dnf. $ sudo yum update Loaded plugins: langpacks fedora/21/x86_64/metalink | 22 kB 00:00:00 fedora | 3.8 kB 00:00:00 updates/21/x86_64/metalink | 25 kB 00:00:00 updates | 4.9 kB 00:00:00 (1/4): fedora/21/x86_64/group_gz | 232 kB 00:00:00 updates/21/x86_64/primary_db FAILED http://mirror.netcologne.de/fedora/linux/updates/21/x86_64/repodata/d07ceea118929066df06bfaa36d42bae37e0ccb26905a8e06470a1c0c9d69c1d-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found Trying other mirror. (2/4): updates/21/x86_64/group_gz | 406 kB 00:00:00 (3/4): updates/21/x86_64/primary_db | 4.2 MB 00:00:00 fedora/21/x86_64/primary_db FAILED - ] 2.5 MB/s | 21 MB 00:00:00 ETA http://mirror.netcologne.de/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 14] curl#18 - "transfer closed with 16119 bytes remaining to read" Trying other mirror. fedora/21/x86_64/primary_db FAILED ==] 1.3 kB/s | 39 MB --:--:-- ETA ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 12] Timeout on ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds') Trying other mirror. fedora/21/x86_64/primary_db FAILED http://mirror.karneval.cz/pub/linux/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable Trying other mirror. fedora/21/x86_64/primary_db FAILED http://ftp.upjs.sk/pub/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable Trying other mirror. fedora/21/x86_64/primary_db FAILED http://mirror.proserve.nl/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable Trying other mirror. fedora/21/x86_64/primary_db FAILED http://mirror.slu.cz/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable Trying other mirror. fedora/21/x86_64/primary_db FAILED ] 68 B/s | 5.7 MB 69:57:39 ETA http://vesta.informatik.rwth-aachen.de/ftp/pub/Linux/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: [Errno 12] Timeout on http://vesta.informatik.rwth-aachen.de/ftp/pub/Linux/fedora/linux/development/21/x86_64/os/repodata/4c0ea0d0ca8fd81fd3a96cacabfbcf9e02c33125670505fcf20aacefab48df02-primary.sqlite.xz: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds') Trying other mirror. (4/4): fedora/21/x86_64/primary_db | 17 MB 00:00:15 (1/2): updates/21/x86_64/updateinfo | 486 kB 00:00:00 updates/21/x86_64/pkgtags FAILED http://mirror.netcologne.de/fedora/linux/updates/21/x86_64/repodata/c0c7a3f6711eaca1a55b75fe7cc31aef35363d87e0b6a57f6fb4cc5b33345493-pkgtags.sqlite.gz: [Errno 14] HTTP Error 404 - Not Found Trying other mirror. (2/2): updates/21/x86_64/pkgtags | 1.3 MB 00:00:00 No packages marked for update
Hi , I am seeing this in fc23 . once fastest mirror gets broken , updating/installing new packages becomes crawls to halt . there shall be some logic in dnf or librepo to downgrade a once broken mirror - speeds are not constant over time - ( timeout setting in dnf does not help too much ) as after a timeout the mirror is not downgraded as being slow . How to test : note fastest mirror and block it in firewall or change in /etc/hosts its hostname to point to an non existing ip , or to a system which has no repo . The actual result is that dnf stil tries to download from broken mirror instead of expected result of trying others .