Bug 1464843 - livemedia-creator: Cannot create live media using repos with metalinks
livemedia-creator: Cannot create live media using repos with metalinks
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
26
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Radek Vykydal
Fedora Extras Quality Assurance
AcceptedBlocker https://fedoraproject...
: CommonBugs, Reopened
: 1464736 (view as bug list)
Depends On:
Blocks: F27FinalBlocker
  Show dependency treegraph
 
Reported: 2017-06-26 00:24 EDT by Neal Gompa
Modified: 2017-11-15 19:23 EST (History)
21 users (show)

See Also:
Fixed In Version: anaconda-27.20.2-1 anaconda-27.20.2-1.fc27
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-05 17:03:33 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Shell script to reproduce issue when building Fedora Workstation 26 (657 bytes, text/plain)
2017-06-26 00:24 EDT, Neal Gompa
no flags Details
anaconda/livemedia-creator dnf.librepo.log (57.48 KB, text/plain)
2017-06-26 00:33 EDT, Neal Gompa
no flags Details
tarball of logs and ks (15.44 KB, application/x-gzip)
2017-06-27 12:11 EDT, Neal Gompa
no flags Details

  None (edit)
Description Neal Gompa 2017-06-26 00:24:28 EDT
Created attachment 1291822 [details]
Shell script to reproduce issue when building Fedora Workstation 26

Description of problem:
When using a kickstart file that has metalinks as sources, something is screwing up and causing it to fail.

Note that metalink sources work fine with DNF itself, but livemedia-creator seems to have problems.

Name-Version-Release number of components:
lorax-26.8-1.fc26.x86_64
lorax-lmc-novirt-26.8-1.fc26.x86_64
lorax-templates-generic-26.8-1.fc26.x86_64
python3-dnf-2.5.1-1.fc26.noarch
libdnf-0.9.1-1.fc26.x86_64
dnf-conf-2.5.1-1.fc26.noarch
dnf-2.5.1-1.fc26.noarch
python3-kickstart-2.35-1.fc26.noarch
pykickstart-2.35-1.fc26.noarch

How reproducible:
Always

Steps to Reproduce:
1. Run attached script

Actual results:
Anaconda dies with a bunch of errors, one of which is that it can't set up any software sources and figure out package selection.

Expected results:
Fedora live media is successfully created.

Additional info:
This seems to affect API consumers of DNF that set up repos through the API, as I believe a similar issue is occurring in livecd-tools, too.
Comment 1 Neal Gompa 2017-06-26 00:33 EDT
Created attachment 1291823 [details]
anaconda/livemedia-creator dnf.librepo.log
Comment 2 Fedora Blocker Bugs Application 2017-06-26 00:45:13 EDT
Proposed as a Blocker for 26-final by Fedora user ngompa using the blocker tracking app because:

 It is currently not possible for someone to reproduce Fedora live media using the kickstarts provided in 'fedora-kickstarts' package, which violates the Kickstarts criteria. (https://fedoraproject.org/wiki/Fedora_26_Final_Release_Criteria#Kickstarts)

In addition, since lorax actually uses Anaconda as the media builder engine, this may mean that metalink repos may not work as sources during a netinstall, which would violate the installer requirement that all valid/supported install sources must work. (https://fedoraproject.org/wiki/Fedora_26_Final_Release_Criteria#Package_and_installer_sources)
Comment 3 Brian Lane 2017-06-26 11:52:07 EDT
This is NOT a blocker. lorax and lmc have never been used with metalinks and have not been designed to work with them. If that's a feature that's desired a RFE needs to be filed (or patches provided).
Comment 4 Adam Williamson 2017-06-26 21:43:10 EDT
To go into a bit of detail on the criteria and how images are created...

The kickstarts from the package are indeed the 'exact kickstart files used to build the release'...*in a sense*. It's a bit of a grey area. What happens is that the compose process takes the exact files from git (which are the ones that wind up in the package), but then does some pre-processing on them before actually using them, including flattening them and adjusting the repository definitions. This is because the compose process works better by using repositories either directly present on the compose machine, or on a local network. It wouldn't make sense to put these repo definitions in the git repo or the package, because you can't use them in really any context besides the Fedora infrastructure, during the compose process.

However, perhaps it *would* make sense to change the kickstarts as they exist in the git repo and the package to use something other than metalinks, if the tools can't actually use metalinks at all. Clearly the real intent behind the criterion is reproducibility, and providing kickstarts that don't work if you try and use them with the tools directly (or with a straightforward ksflatten, at least) isn't helping reproducibility at all. It would clearly be best to aim for the smallest possible level of "required mucking around with the 'source' kickstarts before you can actually build an image". So, it seems reasonable to me to edit this bug, re-open it, and re-assign it to fedora-kickstarts, or just create a new bug or a ticket in the Pagure fedora-kickstarts repo, suggesting not to use metalink lines in the kickstarts if the tools can't work with them (instead we could use a mirrorlist URL, or a download.fp.o URL; whatever the tools can actually work with).
Comment 5 Adam Williamson 2017-06-26 21:45:07 EDT
Oh, forgot to note: if we *do* make that change it should be carefully co-ordinated to make sure that the editing of the kickstart repositories as part of the compose process still works.
Comment 6 Neal Gompa 2017-06-27 02:16:21 EDT
The problem is that this *WORKED BEFORE*!

I was able to use metalinks in Fedora 25 and Fedora 24. The fact that this broke in Fedora 26 is completely unexpected.
Comment 7 Igor Gnatenko 2017-06-27 09:52:00 EDT
You must be kidding....

00:19:03 lr_handle_prepare_mirrorlist: Parsing mirrorlist

I have absolutely no idea whether this is bug in script, lorax or whatever else, but this is not bug in DNF.

THIS IS NOT MIRRORLIST.
Comment 8 Brian Lane 2017-06-27 11:34:56 EDT
Please post the actual kickstart you are using and the logs from anaconda.

As far as I remember metalink/mirrorlist aren't supported with lmc, but I could be forgetting. I always use baseurl to avoid confusion.

Lorax hasn't changed much. Something else changed and broke. Maybe Anaconda?

Hard to tell without:

* Exact kickstart being used.
* ALL the logs from livemedia-creator and anaconda.
Comment 9 Jaroslav Mracek 2017-06-27 11:56:05 EDT
Just to answer question what was changed in universe. The change is in DNF. Originally dnf as well yum was able to use some sort of autodetection for mirrorlist/metalink option. If metalik string was in url, it suggest that it is a metalink, otherwise it used it as mirrorlist. And there is an surprise, it was not working for some cases. Therefore I fix the issue and from that moment we use values as they are provided. If you provide metalink in .repo file we handle it as metalink and same for mirrorlist. Unfortunately after this change anaconda is able to handle only mirrorlists because they do not provide possibility to set metalink yet. Other problem was with EPEL repo where they used mirrorlist=<metalink>. 

I hope that the problem will be solved at places where values are incorrectly set and not on DNF site. But anyway I prepare a backup plan from DNF site (https://github.com/rpm-software-management/dnf/pull/843).
Comment 10 Neal Gompa 2017-06-27 12:11 EDT
Created attachment 1292405 [details]
tarball of logs and ks

@Brian:

I've attached an instance of the logs and the generated kickstart from fedora-kickstarts that is done from the script I attached.
Comment 11 Brian Lane 2017-06-27 14:31:31 EDT
Thanks, I think comment 9 covers it pretty well. DNF broke backwards compatibility and should merge https://github.com/rpm-software-management/dnf/pull/843
Comment 12 Igor Gnatenko 2017-06-27 15:03:57 EDT
(In reply to Brian Lane from comment #11)
> Thanks, I think comment 9 covers it pretty well. DNF broke backwards
> compatibility and should merge
> https://github.com/rpm-software-management/dnf/pull/843

DNF *never ever* had compatibility bug-for-bug with yum.
Comment 13 Jiri Konecny 2017-06-30 10:02:32 EDT
Hello all,

We can fix this in Anaconda by supporting metalinks. That would mean Anaconda will implement `--metalink` feature. Changes will also be required in pykickstart.

This work will be handled when other higher priority items are done. It is too late to fix this bug now for the F26.

Jirka
Comment 14 Stephen Gallagher 2017-06-30 10:38:36 EDT
This doesn't appear to violate the criteria specified and is almost certainly too invasive to permit at this point in the cycle. I'm -1 blocker and -1 FE.
Comment 15 Kamil Páral 2017-06-30 10:57:04 EDT
As I read it, it violates the criteria (the tools used to handle the repo definitions just fine, now they don't, by default it doesn't build, so the repo definition is not correct - not in a sense to be useful for the tools to be used with it). That being said, the workaround is pretty simple and the proper fix is too difficult as stated in comment 13, so I'm -1 F26 blocker, but +1 F27 blocker. This is a valid blocker bug, we're just not able to fix it in a reasonable time. Deferring this to F27 seems fair to me. We'll make sure to include it in CommonBugs.
Comment 16 Mike Ruckman 2017-06-30 11:15:54 EDT
I agree with Kamil. -1 Blocker for F26 and +1 for F27.
Comment 17 Adam Williamson 2017-06-30 15:11:45 EDT
I can go with that, sure. That's four votes, so let's go for it.

BTW, did no-one consider my suggestion to fix this by simply changing the kickstarts to use a download.fp.o URL or a mirrorlist, rather than a metalink?
Comment 18 Neal Gompa 2017-06-30 17:37:59 EDT
Incidentally, the fix proposed in comment 11 was merged upstream, so can we have this backported into F26 stable? Then nothing has to be done about the kickstarts...
Comment 19 Neal Gompa 2017-06-30 17:38:16 EDT
*** Bug 1464736 has been marked as a duplicate of this bug. ***
Comment 20 Neal Gompa 2017-07-02 10:52:30 EDT
(In reply to Neal Gompa from comment #18)
> Incidentally, the fix proposed in comment 11 was merged upstream, so can we
> have this backported into F26 stable? Then nothing has to be done about the
> kickstarts...

That said, I'd much rather have --metalink as described in comment 13. :)
Comment 21 Boyd 2017-07-02 19:09:33 EDT
(In reply to Adam Williamson from comment #17)
> I can go with that, sure. That's four votes, so let's go for it.
> 
> BTW, did no-one consider my suggestion to fix this by simply changing the
> kickstarts to use a download.fp.o URL or a mirrorlist, rather than a
> metalink?

(In reply to Jiri Konecny from comment #13)
> Hello all,
> 
> We can fix this in Anaconda by supporting metalinks. That would mean
> Anaconda will implement `--metalink` feature. Changes will also be required
> in pykickstart.
> 
> This work will be handled when other higher priority items are done. It is
> too late to fix this bug now for the F26.
> 
> Jirka

Whatever thé case is, are you guys willing to skip a Fedora that will barf creating a live image of anything?  All and every Kickstart fails as is.
Comment 22 Neal Gompa 2017-07-02 19:58:28 EDT
(In reply to Boyd from comment #21)
> 
> Whatever thé case is, are you guys willing to skip a Fedora that will barf
> creating a live image of anything?  All and every Kickstart fails as is.

The quick-and-dirty fix is to backport https://github.com/rpm-software-management/dnf/commit/197e13802200523200f779db5a55eb8d3bd3e425 into dnf now for Fedora 26. This will fix both livemedia-creator and livecd-creator (as both leverage the same API constructs for loading repositories). I did this already for Mageia's dnf package to fix livecd-tools, so Fedora should do the same.

Then pykickstart can be fixed for F27 and kickstart files can be adjusted to be explicitly use `--metalink` (it's mind boggling that in the nearly 10 years we've been using metalinks that this hasn't been fixed!).

Then the hack can die in a fire. :)
Comment 23 Adam Williamson 2017-07-03 12:45:19 EDT
"All and every Kickstart fails as is."

Well no, no it doesn't; only ones that specify metalink repo URLs. It's easy enough to tweak them to be straight repo URLs or mirrorlist URLs.
Comment 24 Boyd 2017-07-03 15:11:44 EDT
(In reply to Adam Williamson from comment #23)
> "All and every Kickstart fails as is."
> 
> Well no, no it doesn't; only ones that specify metalink repo URLs. It's easy
> enough to tweak them to be straight repo URLs or mirrorlist URLs.

Yeah its easy enough to 'tweak them'.  But the barfing error messages are very confusiing.  Is it a DNS problem, a DNF problem or what?  Maybe you know but not everyone does. A whole lot of users out there want to make a live CD and have no idea why this is failing.   In that case please make some kind of communication that the kickstart files don't work in F26 and need <tweaking>.
Comment 25 Adam Williamson 2017-07-03 18:26:06 EDT
That's what the CommonBugs tag is for.
Comment 27 Fedora Update System 2017-09-29 12:06:43 EDT
anaconda-27.20.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
Comment 28 Fedora Update System 2017-10-01 20:55:46 EDT
anaconda-27.20.2-1.fc27, pykickstart-2.39-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
Comment 29 Fedora Update System 2017-10-02 06:50:31 EDT
anaconda-27.20.2-1.fc27 pykickstart-2.39-1.fc27 python-simpleline-0.6-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
Comment 30 Fedora Update System 2017-10-02 17:26:03 EDT
anaconda-27.20.2-1.fc27, pykickstart-2.39-1.fc27, python-simpleline-0.6-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6dd3da2595
Comment 31 Fedora Update System 2017-10-05 17:03:33 EDT
anaconda-27.20.2-1.fc27, pykickstart-2.39-1.fc27, python-simpleline-0.6-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 32 Adam Williamson 2017-11-15 19:23:36 EST
Can someone confirm whether this in fact works with F27 Final? I want to know whether to mark the commonbugs issue as 'resolved' or not. Thanks.

Note You need to log in before you can comment on or make changes to this bug.