Created attachment 1291822 [details] Shell script to reproduce issue when building Fedora Workstation 26 Description of problem: When using a kickstart file that has metalinks as sources, something is screwing up and causing it to fail. Note that metalink sources work fine with DNF itself, but livemedia-creator seems to have problems. Name-Version-Release number of components: lorax-26.8-1.fc26.x86_64 lorax-lmc-novirt-26.8-1.fc26.x86_64 lorax-templates-generic-26.8-1.fc26.x86_64 python3-dnf-2.5.1-1.fc26.noarch libdnf-0.9.1-1.fc26.x86_64 dnf-conf-2.5.1-1.fc26.noarch dnf-2.5.1-1.fc26.noarch python3-kickstart-2.35-1.fc26.noarch pykickstart-2.35-1.fc26.noarch How reproducible: Always Steps to Reproduce: 1. Run attached script Actual results: Anaconda dies with a bunch of errors, one of which is that it can't set up any software sources and figure out package selection. Expected results: Fedora live media is successfully created. Additional info: This seems to affect API consumers of DNF that set up repos through the API, as I believe a similar issue is occurring in livecd-tools, too.
Created attachment 1291823 [details] anaconda/livemedia-creator dnf.librepo.log
Proposed as a Blocker for 26-final by Fedora user ngompa using the blocker tracking app because: It is currently not possible for someone to reproduce Fedora live media using the kickstarts provided in 'fedora-kickstarts' package, which violates the Kickstarts criteria. (https://fedoraproject.org/wiki/Fedora_26_Final_Release_Criteria#Kickstarts) In addition, since lorax actually uses Anaconda as the media builder engine, this may mean that metalink repos may not work as sources during a netinstall, which would violate the installer requirement that all valid/supported install sources must work. (https://fedoraproject.org/wiki/Fedora_26_Final_Release_Criteria#Package_and_installer_sources)
This is NOT a blocker. lorax and lmc have never been used with metalinks and have not been designed to work with them. If that's a feature that's desired a RFE needs to be filed (or patches provided).
To go into a bit of detail on the criteria and how images are created... The kickstarts from the package are indeed the 'exact kickstart files used to build the release'...*in a sense*. It's a bit of a grey area. What happens is that the compose process takes the exact files from git (which are the ones that wind up in the package), but then does some pre-processing on them before actually using them, including flattening them and adjusting the repository definitions. This is because the compose process works better by using repositories either directly present on the compose machine, or on a local network. It wouldn't make sense to put these repo definitions in the git repo or the package, because you can't use them in really any context besides the Fedora infrastructure, during the compose process. However, perhaps it *would* make sense to change the kickstarts as they exist in the git repo and the package to use something other than metalinks, if the tools can't actually use metalinks at all. Clearly the real intent behind the criterion is reproducibility, and providing kickstarts that don't work if you try and use them with the tools directly (or with a straightforward ksflatten, at least) isn't helping reproducibility at all. It would clearly be best to aim for the smallest possible level of "required mucking around with the 'source' kickstarts before you can actually build an image". So, it seems reasonable to me to edit this bug, re-open it, and re-assign it to fedora-kickstarts, or just create a new bug or a ticket in the Pagure fedora-kickstarts repo, suggesting not to use metalink lines in the kickstarts if the tools can't work with them (instead we could use a mirrorlist URL, or a download.fp.o URL; whatever the tools can actually work with).
Oh, forgot to note: if we *do* make that change it should be carefully co-ordinated to make sure that the editing of the kickstart repositories as part of the compose process still works.
The problem is that this *WORKED BEFORE*! I was able to use metalinks in Fedora 25 and Fedora 24. The fact that this broke in Fedora 26 is completely unexpected.
You must be kidding.... 00:19:03 lr_handle_prepare_mirrorlist: Parsing mirrorlist I have absolutely no idea whether this is bug in script, lorax or whatever else, but this is not bug in DNF. THIS IS NOT MIRRORLIST.
Please post the actual kickstart you are using and the logs from anaconda. As far as I remember metalink/mirrorlist aren't supported with lmc, but I could be forgetting. I always use baseurl to avoid confusion. Lorax hasn't changed much. Something else changed and broke. Maybe Anaconda? Hard to tell without: * Exact kickstart being used. * ALL the logs from livemedia-creator and anaconda.
Just to answer question what was changed in universe. The change is in DNF. Originally dnf as well yum was able to use some sort of autodetection for mirrorlist/metalink option. If metalik string was in url, it suggest that it is a metalink, otherwise it used it as mirrorlist. And there is an surprise, it was not working for some cases. Therefore I fix the issue and from that moment we use values as they are provided. If you provide metalink in .repo file we handle it as metalink and same for mirrorlist. Unfortunately after this change anaconda is able to handle only mirrorlists because they do not provide possibility to set metalink yet. Other problem was with EPEL repo where they used mirrorlist=<metalink>. I hope that the problem will be solved at places where values are incorrectly set and not on DNF site. But anyway I prepare a backup plan from DNF site (https://github.com/rpm-software-management/dnf/pull/843).
Created attachment 1292405 [details] tarball of logs and ks @Brian: I've attached an instance of the logs and the generated kickstart from fedora-kickstarts that is done from the script I attached.
Thanks, I think comment 9 covers it pretty well. DNF broke backwards compatibility and should merge https://github.com/rpm-software-management/dnf/pull/843
(In reply to Brian Lane from comment #11) > Thanks, I think comment 9 covers it pretty well. DNF broke backwards > compatibility and should merge > https://github.com/rpm-software-management/dnf/pull/843 DNF *never ever* had compatibility bug-for-bug with yum.
Hello all, We can fix this in Anaconda by supporting metalinks. That would mean Anaconda will implement `--metalink` feature. Changes will also be required in pykickstart. This work will be handled when other higher priority items are done. It is too late to fix this bug now for the F26. Jirka
This doesn't appear to violate the criteria specified and is almost certainly too invasive to permit at this point in the cycle. I'm -1 blocker and -1 FE.
As I read it, it violates the criteria (the tools used to handle the repo definitions just fine, now they don't, by default it doesn't build, so the repo definition is not correct - not in a sense to be useful for the tools to be used with it). That being said, the workaround is pretty simple and the proper fix is too difficult as stated in comment 13, so I'm -1 F26 blocker, but +1 F27 blocker. This is a valid blocker bug, we're just not able to fix it in a reasonable time. Deferring this to F27 seems fair to me. We'll make sure to include it in CommonBugs.
I agree with Kamil. -1 Blocker for F26 and +1 for F27.
I can go with that, sure. That's four votes, so let's go for it. BTW, did no-one consider my suggestion to fix this by simply changing the kickstarts to use a download.fp.o URL or a mirrorlist, rather than a metalink?
Incidentally, the fix proposed in comment 11 was merged upstream, so can we have this backported into F26 stable? Then nothing has to be done about the kickstarts...
*** Bug 1464736 has been marked as a duplicate of this bug. ***
(In reply to Neal Gompa from comment #18) > Incidentally, the fix proposed in comment 11 was merged upstream, so can we > have this backported into F26 stable? Then nothing has to be done about the > kickstarts... That said, I'd much rather have --metalink as described in comment 13. :)
(In reply to Adam Williamson from comment #17) > I can go with that, sure. That's four votes, so let's go for it. > > BTW, did no-one consider my suggestion to fix this by simply changing the > kickstarts to use a download.fp.o URL or a mirrorlist, rather than a > metalink? (In reply to Jiri Konecny from comment #13) > Hello all, > > We can fix this in Anaconda by supporting metalinks. That would mean > Anaconda will implement `--metalink` feature. Changes will also be required > in pykickstart. > > This work will be handled when other higher priority items are done. It is > too late to fix this bug now for the F26. > > Jirka Whatever thé case is, are you guys willing to skip a Fedora that will barf creating a live image of anything? All and every Kickstart fails as is.
(In reply to Boyd from comment #21) > > Whatever thé case is, are you guys willing to skip a Fedora that will barf > creating a live image of anything? All and every Kickstart fails as is. The quick-and-dirty fix is to backport https://github.com/rpm-software-management/dnf/commit/197e13802200523200f779db5a55eb8d3bd3e425 into dnf now for Fedora 26. This will fix both livemedia-creator and livecd-creator (as both leverage the same API constructs for loading repositories). I did this already for Mageia's dnf package to fix livecd-tools, so Fedora should do the same. Then pykickstart can be fixed for F27 and kickstart files can be adjusted to be explicitly use `--metalink` (it's mind boggling that in the nearly 10 years we've been using metalinks that this hasn't been fixed!). Then the hack can die in a fire. :)
"All and every Kickstart fails as is." Well no, no it doesn't; only ones that specify metalink repo URLs. It's easy enough to tweak them to be straight repo URLs or mirrorlist URLs.
(In reply to Adam Williamson from comment #23) > "All and every Kickstart fails as is." > > Well no, no it doesn't; only ones that specify metalink repo URLs. It's easy > enough to tweak them to be straight repo URLs or mirrorlist URLs. Yeah its easy enough to 'tweak them'. But the barfing error messages are very confusiing. Is it a DNS problem, a DNF problem or what? Maybe you know but not everyone does. A whole lot of users out there want to make a live CD and have no idea why this is failing. In that case please make some kind of communication that the kickstart files don't work in F26 and need <tweaking>.
That's what the CommonBugs tag is for.
https://github.com/rhinstaller/pykickstart/pull/172 https://github.com/rhinstaller/anaconda/pull/1169
anaconda-27.20.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
anaconda-27.20.2-1.fc27, pykickstart-2.39-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
anaconda-27.20.2-1.fc27 pykickstart-2.39-1.fc27 python-simpleline-0.6-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
anaconda-27.20.2-1.fc27, pykickstart-2.39-1.fc27, python-simpleline-0.6-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
anaconda-27.20.2-1.fc27, pykickstart-2.39-1.fc27, python-simpleline-0.6-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Can someone confirm whether this in fact works with F27 Final? I want to know whether to mark the commonbugs issue as 'resolved' or not. Thanks.