Description of problem: For years I've been seeing issues with Fedora mirrors - dnf not able to get its data files. I understand the issues are most likely with mirroring process, but not sure where to report that. Hopefully someone can reassign or bring to the attention of the right people. Version-Release number of selected component (if applicable): dnf-4.2.2-2.fc30.noarch How reproducible: Sometimes, but often, and has been going on for years. 1 - There is a local (Russia) Fedora mirror, with Yandex, a large internet company. Very fast for me. 2 - University of Chemnitz in Germany is a Tier 1 mirror. Watch what just happened with baseurl=https://mirror.yandex.ru/fedora/linux/updates/testing/$releasever/Modular/$basearch/ <tab> https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/$releasever/Modular/$basearch/ ----- repo: downloading from remote: updates-testing-modular error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck). error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/a6f6738f280a3ab16d40544c80d0169d1a34f91d0f46a1ac4d2463f68996163d-primary.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/a6f6738f280a3ab16d40544c80d0169d1a34f91d0f46a1ac4d2463f68996163d-primary.xml.zck). error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/30/Modular/x86_64/repodata/a6f6738f280a3ab16d40544c80d0169d1a34f91d0f46a1ac4d2463f68996163d-primary.xml.zck). error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck). ---- Issue 1 - with Yandex "Server returned nothing (no headers, no data)" I can do curl for the url given by dnf and everything appears fine: curl -v https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck * Trying 213.180.204.183... * TCP_NODELAY set * Connected to mirror.yandex.ru (213.180.204.183) port 443 (#0) < HTTP/1.1 200 OK < Server: nginx/1.8.1 < Date: Sat, 04 May 2019 10:15:22 GMT < Content-Type: application/octet-stream < Content-Length: 2142638 < Last-Modified: Fri, 03 May 2019 04:19:01 GMT < Connection: keep-alive < ETag: "5ccbc135-20b1ae" < Accept-Ranges: bytes So why is dnf having trouble? Issue 2 - dnf then tries Univ. Chemnitz, and gets corrupted data error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/30/Modular/x86_64/repodata/a6f6738f280a3ab16d40544c80d0169d1a34f91d0f46a1ac4d2463f68996163d-primary.xml.zck). error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck). This second issue looks like either corrupted files, or dnf not checking the checksum right, or something, but point it the files are there but dnf can't use them. curl -v for the "filelists.xml.zck" also appears to work, Content-Length: 2142638 (same as we got from Yandex using curl). But dnf is not happy. And rememeber that Chemnitz is a Tier 1 mirror, supposed to have everything "100% perfect". -------- Finally using metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-modular-f$releasever&arch=$basearch does work, presumably there are more mirrors (but I can't see them even with "v"). Bottom line - something weird going on with Fedora mirrors and has been for years (in my experience). How to bring this to the attention of the right people?
On second issue ("error: Zchunk header checksum didn't match expected checksum"): I tried fetching 0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck from both Yandex and Chemnitz and comparing what I get. The files from both sources are identical. I am also able to use zstdless on them (although the content appears to be binary). ----- wget -O filelists-yandex.xml.zck https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck wget -O filelists-chemnitz.xml.zck https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck If they came from Fedora's origin server - and dnf says that "error: Zchunk header checksum didn't match expected checksum" - then the origin file is bad?
Created attachment 1563134 [details] file from yandex
Created attachment 1563135 [details] file from chemintz
One more thing. I installed "zchunk" command lines tools - and am able to decompress and view the attached .xml.zck files. And yet dnf complained about "Zchunk header checksum didn't match expected checksum". Looks to me like potentially there is an issue here in dnf itself as well.
Adding zchunk=False to dnf.conf made both mirrors (Yandex and Chemnitz) work flawlessly. I've done "dnf clean all / dnf --refresh update" multiple times, no errors. So there are two issues with zchunk format files. And it does not look like it's the web servers, since I'm able to fetch with curl and get identical content which can be decompressed / viewed with "unzck". To summarize better, the two issues are: 1 - Yandex error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck). Server: nginx/1.8.1 2 - Chemnitz error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/testing/30/Modular/x86_64/repodata/0b0804a49c5ec3aece8ef73a5a4ce1b1b93a03d093d918d6944e7592ab084a41-filelists.xml.zck). Server: Apache/2.2.34 (Scientific Linux)
Moving over to zchunk component (although it may likely be something in the librepo zchunk support)
This is definitely a problem with zchunk (technically, the problem is more with how these web servers are configured handle ranges, but zchunk should still degrade to downloading the whole file rather than erroring out). I've just returned from the States, so it may be a few days before I get a chance to look at this in more detail, but I will get it sorted out ASAP.
(In reply to Jonathan Dieter from comment #7) > zchunk should still degrade to downloading the whole file +1 to that, no +100500! > the problem is more with how these web servers are configured handle ranges Are you willing / able to share more info? "curl -r" to get a range did work for me (on these servers), so I'm wondering what is wrong.
(In reply to Kostya Vasilyev from comment #8) > (In reply to Jonathan Dieter from comment #7) > > > zchunk should still degrade to downloading the whole file > > +1 to that, no +100500! > > > the problem is more with how these web servers are configured handle ranges > > Are you willing / able to share more info? > > "curl -r" to get a range did work for me (on these servers), so I'm > wondering what is wrong. Yes, but I'm extremely jet-lagged at the moment and have only done an initial investigation. When there are multiple ranges that zchunk needs to download, it will combine them into the largest number of ranges that the web server supports in a single request. Most servers support a fairly large number of ranges (256 or 512) in a single request, but some have been configured to support far fewer. If I remember correctly, most web servers will respond with a 200 response rather than a 206 if you request too many ranges, and zchunk should handle a 200 response correctly. I believe these two sites are giving a different response (though I haven't yet traced the curl requests that zchunk is using). FWIW, I can reproduce the bug using just the zchunk tools when using zckdl -s <old version of file> <url of new version of file>. However, depending on what's actually going on, we may have to push fixes to both zchunk and librepo.
Ok, so I've looked at this in more detail, and, unfortunately, it's not an easy fix. First off, the Chemnitz mirror seems to be fine for me. I've just downloaded both primary.zck and filelists.zck with over 217 missing chunks and it works fine. It's possible that there's a bug in how dnf's zchunk code handles > 256 separate download ranges, but I don't think that's likely. If you can reproduce, please send me the source file (/var/cache/dnf/updates-testing-modular-*/*filelists.xml.zck). Yandex is just messed up. When it receives more than a few ranges in a single request, it just closes the connection rather than returning a 200 (the full file) or 416 (invalid range request). Zchunk does have a drop-off in the number of ranges that it tries to download in a single request (255, 127, 7, 2, 1). After patching zckdl to recognize a closed connection as a possible range failure, Yandex then added in a significant delay between the two range download attempt and the single range download attempt. This means that, even after fixing zckdl, Yandex still takes a ridiculously long time to download. So how do we fix this? Option 1: Leave everything as is. When Yandex is used as part of a mirrorlink and is downloading multiple ranges, it will quickly fail, and dnf will move on to the next mirror. If Yandex is in the baseurl, dnf will fail unless zchunk=False is set. Option 2: Update librepo to treat a closed connection as a range failure. When Yandex is used as part of a mirrorlink and is downloading multiple ranges, it will work very slowly, but it will stay with the Yandex mirror. If Yandex is in the baseurl, dnf will work very slowly, but will succeed. Option 3: Fix librepo so that it only treats a closed connection as a range failure if it happens with a baseurl or a single mirror. Yandex as a mirrorlink will quickly fail to the next mirror, but Yandex in the baseurl will work correctly (if slowly). This option will take the longest time to code and test, but we could fix the whole "DNF doesn't retry downloads" as part of the patchset. Thoughts?
Thanks for the analysis. 1 - On Chemnitz I enabled zchunk in DNF config and then did: $ sudo dnf clean all $ sudo dnf --refresh -v update Which got me this, the expected Yandex errors but also same old Chemnitz "checksum" error. The repository was updates-modular (not updates-testing-modular). I'm attaching a tar of all of (/var/cache/dnf/updates-modular-*/. ----- error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/5147d813f6325fafb29b73e5281d499bb652824486ceaf20f9938413240719de-primary.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/5147d813f6325fafb29b73e5281d499bb652824486ceaf20f9938413240719de-primary.xml.zck). error: Curl error (52): Server returned nothing (no headers, no data) for https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/39349f1238e6a7f7cd0487bb92eb1da66bdde28c0abbf86a57f6e8adbf1c4dcd-filelists.xml.zck [Empty reply from server] (https://mirror.yandex.ru/fedora/linux/updates/30/Modular/x86_64/repodata/39349f1238e6a7f7cd0487bb92eb1da66bdde28c0abbf86a57f6e8adbf1c4dcd-filelists.xml.zck). error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/30/Modular/x86_64/repodata/5147d813f6325fafb29b73e5281d499bb652824486ceaf20f9938413240719de-primary.xml.zck). error: Zchunk header checksum didn't match expected checksum (https://ftp.tu-chemnitz.de/pub/linux/fedora/linux/updates/30/Modular/x86_64/repodata/39349f1238e6a7f7cd0487bb92eb1da66bdde28c0abbf86a57f6e8adbf1c4dcd-filelists.xml.zck). ---- My fedora*.repo's are set like this: baseurl=https://mirror.yandex.ru/... https://ftp.tu-chemnitz.de/.... My whole dnf.conf [main] gpgcheck=1 installonly_limit=3 clean_requirements_on_remove=True max_parallel_downloads=10 deltarpm=False zchunk=True # for the test, or False normally 2 - Yandex I think the best solution is #3 with "if it's the LAST mirror to be tried" - meaning it will also work if it's a single baseurl listed or the last baseurl item listed or the only or last url returned by basemeta. 3 - Yandex They're an official mirror, some way for someone to contact them and ask to fix (update) the server?
Created attachment 1566415 [details] /var/cache/dnf/updates-modular-*
Also on Yandex - as I understand, it gets ridiculously slow because of the drop-off in the number of chunks, right? Since a connection close is an invalid response - can dnf avoid the drop-off in this case and go straight to full file fetch? Then it would still work and maybe not so slowly - even without the "if LAST" logic?
(In reply to Kostya Vasilyev from comment #11) > Thanks for the analysis. > > 1 - On Chemnitz > > I enabled zchunk in DNF config and then did: > > $ sudo dnf clean all > $ sudo dnf --refresh -v update > > Which got me this, the expected Yandex errors but also same old Chemnitz > "checksum" error. Remove Yandex and see if that fixes the problem. If it does, it's a bug in dnf, but one that should be relatively easy to fix. As for the "best" solution, I believe that solution 3 is superior to the others, but it's going to be pretty invasive and I have no idea when I'll have time to implement it. Yandex gets slow because it basically freezes the connection for over 30 seconds when it checks using a double-range in a request. As for going straight to the full file fetch on an invalid response, it makes more sense to just move to the next (non-broken) mirror. I realize this isn't working for you right now, but that's something that can be fixed.
Created attachment 1566423 [details] /var/cache/dnf
> Remove Yandex and see if that fixes the problem. If it does, it's a bug in dnf, but one that should be relatively easy to fix. I removed yandex keeping only chemnitz it worked with zchunk enabled (tested with two repositories). > As for going straight to the full file fetch on an invalid response, it makes more sense to just move to the next (non-broken) mirror. But if there is no "next" mirror? I guess for most people there would be - if they don't touch their repo files and they're set to "meta". Even for Russian users there are mirrors besides Yandex, but Yandex is probably the first one returned. Why not treat "server closed connection in response to range request" as a "hard" failure - and move on to full fetch immediately without trying another range request (with smaller number of ranges)? As I understand the only valid responses to a range request are 200 and 416 and server closing the connection definitely isn't. BTW - I know how hard it can be to first troubleshoot then find acceptable workarounds when "something else" you're dealing with is broken. > I realize this isn't working for you right now, but that's something that can be fixed. Well what worked for me is turning off zchunk in dnf.conf. Please whatever you do - don't remove that setting.
(In reply to Kostya Vasilyev from comment #16) > > Remove Yandex and see if that fixes the problem. If it does, it's a bug in dnf, but one that should be relatively easy to fix. > > I removed yandex keeping only chemnitz it worked with zchunk enabled (tested > with two repositories). > > > As for going straight to the full file fetch on an invalid response, it makes more sense to just move to the next (non-broken) mirror. > > But if there is no "next" mirror? > > I guess for most people there would be - if they don't touch their repo > files and they're set to "meta". Even for Russian users there are mirrors > besides Yandex, but Yandex is probably the first one returned. If there is no next mirror, then we'd want to download the full file. Handling the last mirror differently than previous mirrors isn't currently supported in librepo, so this code needs to be written. > Why not treat "server closed connection in response to range request" as a > "hard" failure - and move on to full fetch immediately without trying > another range request (with smaller number of ranges)? What I'm saying is that we are treating it as a "hard" failure... and moving on to the next mirror. I do agree that, once we get the "last mirror" code in, a full fetch makes more sense in this situation than trying the range requests. > As I understand the only valid responses to a range request are 200 and 416 > and server closing the connection definitely isn't. > > BTW - I know how hard it can be to first troubleshoot then find acceptable > workarounds when "something else" you're dealing with is broken. > > > I realize this isn't working for you right now, but that's something that can be fixed. > > Well what worked for me is turning off zchunk in dnf.conf. Please whatever > you do - don't remove that setting. That setting isn't going anywhere. It's the emergency override, and, at least as far as I'm concerned, will be a part of dnf until either dnf or zchunk are replaced with something else.
> What I'm saying is that we are treating it as a "hard" failure... and moving on to the next mirror. I do agree that, once we get the "last mirror" code in, a full fetch makes more sense in this situation than trying the range requests. OK, so for Yandex the only possible improvement would rely on differentiating last/non-last mirror - and I understand there are difficulties with that. But seems there is still another bug. baseurl=chemnitz works. But baseurl=yandex chemnitz after yandex fails, and dnf moves on to chemnitz, then chemnitz fail with bad checksums. It looks to me like maybe dnf code isn't fully clearing some internal zchunk-related state when moving from url to url. I think that's what you wanted me to test here: --- > Remove Yandex and see if that fixes the problem. If it does, it's a bug in dnf, but one that should be relatively easy to fix. I removed yandex keeping only chemnitz it worked with zchunk enabled (tested with two repositories). ---
> OK, so for Yandex the only possible improvement would rely on differentiating last/non-last mirror - and I understand there are difficulties with that. Scratch that. After thinking about it some more... It would - maybe - be an improvement to handle yandex error as "ok let's retry without ranges right away". But it's not clear to me if it's really better than "just skip to next mirror". I would just leave that alone then. The real improvement for Yandex would be to get them to fix their server. Somehow. The team that handles mirroring presumably has a way to contact Yandex mirror service. I hope. And I hope someone can try this. ------- Now there is also a real bug - I believe - in dnf, just to mention it once more. And it is: - if dnf skips from yandex to chemnitz, then there are checksum errors from chemnitz - if chemnitz is the only baseurl then everything is fine - presumably some code in dnf doesn't fully clear its zchunk client state when skipping from one url (yandex) to next (chemnitz)
Suppose, this is global problem now... I Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-f30&arch=x86_64': Checksum error /var/cache/dnf/updates-modular-783da5de2e38c644/repodata/f850ebe08c1f80a08866f62242b0e6a7091e500ef2a205bd3dc1c16640f06f1c-primary.xml.zck: Unable to read zchunk lead. I can't upgrade from any servers without zchunk=False
(In reply to olegon.ru from comment #20) > Suppose, this is global problem now... I > > > Cannot download > 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular- > f30&arch=x86_64': Checksum error > /var/cache/dnf/updates-modular-783da5de2e38c644/repodata/ > f850ebe08c1f80a08866f62242b0e6a7091e500ef2a205bd3dc1c16640f06f1c-primary.xml. > zck: Unable to read zchunk lead. > > I can't upgrade from any servers without zchunk=False Thanks for checking this. Can you please attach /var/log/dnf.librepo.log after you've cleared your cache and set zchunk=True?
Created attachment 1567022 [details] log echo >dnf.librepo.log dnf clean all dnf --refresh upgrade
(In reply to olegon.ru from comment #22) > Created attachment 1567022 [details] > log > > echo >dnf.librepo.log > dnf clean all > dnf --refresh upgrade That's exactly what I needed. I just needed to verify that yandex was your first mirror. Your report has just upped the severity of this bug (because it's presumably affecting anyone who gets yandex as their first mirror), so I'll try to get a fix done today.
Yandex mirrors need to be completely disabled for all Fedora users due to their internal instability and inconsistent condition.
Yandex is probably first choice (returned by download.fedora) for anyone in Russia. And other than its range issue and a dnf bug (where it gets corrupted checksum on next url) its local and fast. There is another mirror its f30 directories are often empty. http://mirror.linux-ia64.org/fedora/linux/updates/30/Everything/x86_64/repodata/ This only leaves rbc.ru but I had trouble with it before. Maybe it's better now not sure.
rbc - does not have Fedora 30 http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/ This means that Yandex is the only semi-working mirror (only with zchunk=False) in Russia at this time. - rbc.ru - does not have Fedora 30 - linux-ia64.org - missing files, maybe stopped mirroring
Ok, I've fixed the main issue here, which is: if the first mirror fails part-way through the download, all further mirrors will fail. There's a pull request at https://github.com/rpm-software-management/librepo/pull/151 with further information. I've created a scratch build of the fix that should be available for a while: https://koji.fedoraproject.org/koji/taskinfo?taskID=34789926 If you want to test right now, go ahead and run: sudo dnf update https://kojipkgs.fedoraproject.org//work/tasks/9926/34789926/librepo-1.9.6-3.fc30.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/9926/34789926/python3-librepo-1.9.6-3.fc30.x86_64.rpm You may have to set zchunk=False in dnf.conf to run this command. Please set it back to True to verify that everything's working as it should. Please note that if you have only *one* mirror set and it's Yandex, it will still fail.
librepo-1.9.6-3.fc30.x86_64.rpm works for me: baseurl=yandex chemnitz Still get "server returned nothing" from Yandex, as expected. But then the next url - chemnitz - works. Overall, dnf clean all / dnf update now does run to completion. PS - my system did not have python3-librepo, so did not update.
(In reply to Kostya Vasilyev from comment #28) > librepo-1.9.6-3.fc30.x86_64.rpm works for me: > > baseurl=yandex > chemnitz > > Still get "server returned nothing" from Yandex, as expected. > > But then the next url - chemnitz - works. > > Overall, dnf clean all / dnf update now does run to completion. > > PS - my system did not have python3-librepo, so did not update. Thanks so much for checking this! This is the desired result.
This is fundamentally a librepo issue, and needs to be fixed by updating to librepo 1.10.2.
https://bugzilla.redhat.com/show_bug.cgi?id=1706627
*** Bug 1709478 has been marked as a duplicate of this bug. ***
*** Bug 1710492 has been marked as a duplicate of this bug. ***
librepo-1.10.2-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1c6cbddc8a
librepo-1.10.2-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-28516f8c38
librepo-1.10.2-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1c6cbddc8a
librepo-1.10.2-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-28516f8c38
librepo-1.10.2-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
FEDORA-2019-0f3d77c500 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0f3d77c500
librepo-1.10.2-2.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0f3d77c500
librepo-1.10.2-2.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
dnf install librepo Last metadata expiration check: 0:00:47 ago on Thu 10 Oct 2019 12:02:06 BST. Package librepo-1.10.5-1.fc29.x86_64 is already installed. Yesterday I tried to upgrade from Fed29 to Fed30. It failed. Digging into dnf.log this seems to be rearing its ugly head again. /etc/dnf/dnf.log File “/usr/lib/python3.7/site-packages/dnf/base.py”, line 136, in _add_repo_to_sack repo.load() File “/usr/lib/python3.7/site-packages/dnf/repo.py”, line 558, in load raise dnf.exceptions.RepoError(str(e)) dnf.exceptions.RepoError: Failed to synchronize cache for repo ‘fedora-modular’ 2019-10-10T02:31:48Z CRITICAL Error: Failed to synchronize cache for repo ‘fedora-modular’ 2019-10-10T03:31:58Z INFO — logging initialized — maven-resolver-util noarch 1:1.1.1-2.module_f28+3939+dc18cd75 fedora-modular 147 k maven-shared-utils noarch 3.2.1-0.1.module_f28+3939+dc18cd75 fedora-modular 164 k maven-wagon-file noarch 3.1.0-1.module_f28+3939+dc18cd75 fedora-modular 25 k maven-wagon-http noarch 3.1.0-1.module_f28+3939+dc18cd75 fedora-modular 26 k maven-wagon-http-shared noarch 3.1.0-1.module_f28+3939+dc18cd75 fedora-modular 48 k maven-wagon-provider-api noarch 3.1.0-1.module_f28+3939+dc18cd75 fedora-modular 62 k plexus-cipher noarch 1.7-14.module_f28+3939+dc18cd75 fedora-modular 28 k plexus-classworlds noarch 2.5.2-9.module_f28+3939+dc18cd75 fedora-modular 64 k plexus-interpolation noarch 1.22-9.module_f28+3939+dc18cd75 fedora-modular 78 k regexp noarch 1:1.5-26.module_f28+3872+5b76729e fedora-modular 52 k slf4j noarch 1.7.25-4.module_f28+3939+dc18cd75 fedora-modular 76 k xml-commons-apis noarch 1.4.01-25.module_f28+3872+5b76729e fedora-modular 233 k xz-java noarch 1.8-2.module_f28+3872+5b76729e fedora-modular 105 k system-config-printer x86_64 1.5.11-18.fc30 updates 309 k
of course the was from /var/log/dnf.log
Can you please clear /var/log/dnf.librepo.log, attempt to re-run the upgrade, and then attach /var/log/dnf.librepo.log. If it's empty, set debuglevel=3 in /etc/dnf/dnf.conf and run it again. I can't investigate further without a valid /var/log/dnf.librepo.log.
dnf.log 2019-10-13T17:37:00Z INFO --- logging initialized --- 2019-10-13T17:37:00Z DDEBUG timer: config: 29 ms 2019-10-13T17:37:01Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, $ 2019-10-13T17:37:01Z DEBUG DNF version: 4.2.5 2019-10-13T17:37:01Z DDEBUG Command: dnf system-upgrade reboot 2019-10-13T17:37:01Z DDEBUG Installroot: / 2019-10-13T17:37:01Z DDEBUG Releasever: 29 2019-10-13T17:37:01Z DEBUG cachedir: /var/cache/dnf 2019-10-13T17:37:01Z DDEBUG Base command: system-upgrade 2019-10-13T17:37:01Z DDEBUG Extra commands: ['system-upgrade', 'reboot'] 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-cisco-openh264.repo; Configuration: OptionBinding with id "failovermetho$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermeth$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermeth$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermeth$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "fail$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "fail$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "fail$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermeth$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermeth$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermeth$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exi$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exi$ 2019-10-13T17:37:01Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exi$ 2019-10-13T17:37:01Z DDEBUG Cleaning up. librepo.log 2019-10-13T17:37:00Z DEBUG Librepo version: 1.10.5 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.61.1 OpenSSL/1.1.1d zlib/1.2.11 brotli/1.0.5 libidn2/2.2.0 libpsl/0.20.2 (+libidn2$ 2019-10-13T17:37:00Z DEBUG Current date: 2019-10-13T18:37:00+0100
I'm afraid that still isn't enough detail from dnf.librepo.log. Can you please set debuglevel=3 in /etc/dnf/dnf.conf and try again?
I've just spun up a F29 system, fully updated it, and done a sysupgrade up to F30 through the download stage with no problems. I also verified that the F29 sysupgrade process does not actually use zchunk, which means the problem you're running into is most likely unrelated to this one. Can you please open a new ticket describing your problem (please make sure to include the full logs for both dnf.log and dnf.librepo.log with the debugging option I asked you to turn on).
OK, done : #1761251