Bug 1026834 - method= combined with 'url' in kickstart fails to find packages, in certain situations [NEEDINFO]
method= combined with 'url' in kickstart fails to find packages, in certain s...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
20
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Martin Kolman
Fedora Extras Quality Assurance
RejectedBlocker AcceptedFreezeException
:
Depends On:
Blocks: F20FinalFreezeException 1040138
  Show dependency treegraph
 
Reported: 2013-11-05 09:02 EST by Kamil Páral
Modified: 2013-12-13 00:34 EST (History)
10 users (show)

See Also:
Fixed In Version: anaconda-20.25.15-1.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1040138 (view as bug list)
Environment:
Last Closed: 2013-12-13 00:34:19 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
kparal: needinfo? (jantill)


Attachments (Terms of Use)
anaconda failure screenshot (134.32 KB, image/png)
2013-11-05 09:02 EST, Kamil Páral
no flags Details
ks.cfg (666 bytes, text/plain)
2013-11-05 09:05 EST, Kamil Páral
no flags Details
anaconda.log (with method=) (7.54 KB, text/plain)
2013-11-05 09:05 EST, Kamil Páral
no flags Details
anaconda-yum.conf (with method=) (269 bytes, text/plain)
2013-11-05 09:05 EST, Kamil Páral
no flags Details
packaging.log (with method=) (75.58 KB, text/plain)
2013-11-05 09:05 EST, Kamil Páral
no flags Details
program.log (with method=) (27.87 KB, text/plain)
2013-11-05 09:05 EST, Kamil Páral
no flags Details
storage.log (with method=) (100.52 KB, text/plain)
2013-11-05 09:06 EST, Kamil Páral
no flags Details
syslog (with method=) (63.91 KB, text/plain)
2013-11-05 09:06 EST, Kamil Páral
no flags Details
screenshot with error in TC3 (172.24 KB, image/png)
2013-11-28 08:39 EST, Kamil Páral
no flags Details
screenshot with error in TC4 (109.12 KB, image/png)
2013-12-03 11:08 EST, Kamil Páral
no flags Details

  None (edit)
Description Kamil Páral 2013-11-05 09:02:22 EST
Description of problem:
This command fails:

$ virt-install --name FedoraTest --ram 1024 --disk pool=default,size=10 --location http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/ --initrd-inject ks.cfg --extra-args "ks=file:/ks.cfg"

because Anaconda complains that it can't find a particular package in a repository. It is looking for device-mapper-libs-1.02.82-3.fc20.x86_64.rpm, which is available in the target location:
http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/Packages/d/

However, the included kickstart only enables "fedora" and "updates" repositories, which currently contain only device-mapper-libs-1.02.81-1.fc20.x86_64.rpm:
http://dl.fedoraproject.org/pub/fedora/linux/development/20/x86_64/os/Packages/d/

It seems that anaconda retrieves yum metadata from http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/, but then it tries to use that metadata with standard fedora+updates repos as defined in the kickstart. That obviously doesn't work. The --location metadata must be flushed first, and only then fedora+updates repos should be queried and used.

I don't need to use virt-install, I can simulate exactly the same problem in virt-manager, if I use direct kernel boot (kernel+initrd) or netinst and provide these boot arguments:
method=http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg


However, it is mostly interesting that these boot arguments *work well*:
inst.repo=http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

In another words, changing method= to inst.repo= fixes the situation and correct metadata are used. I have confirmed that in several attempts.

Please fix method= to work exactly same as inst.repo=.


Version-Release number of selected component (if applicable):
anaconda 20.25.5-1
F20 Beta RC2

How reproducible:
always

Steps to Reproduce:
1. boot anaconda using direct kernel boot or netinst
2. use the attached kickstart (http or initrd injection delivery method)
3. add boot argument: method=http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/
4. see that the installation fails
5. replace the boot argument with: inst.repo=http://dl.fedoraproject.org/pub/alt/stage/20-Beta-RC2/Fedora/x86_64/os/
6. see that the installation succeeds

Please note that the above reproducer will only work until device-mapper-libs-1.02.82-3.fc20.x86_64.rpm (and probably some other offending packages) are pushed to stable repositories. Some required packages must be newer in the method= location than in standard fedora repos in order to trigger this issue.
Comment 1 Kamil Páral 2013-11-05 09:02:54 EST
Created attachment 819775 [details]
anaconda failure screenshot
Comment 2 Kamil Páral 2013-11-05 09:05:10 EST
Created attachment 819776 [details]
ks.cfg
Comment 3 Kamil Páral 2013-11-05 09:05:25 EST
Created attachment 819777 [details]
anaconda.log (with method=)
Comment 4 Kamil Páral 2013-11-05 09:05:35 EST
Created attachment 819778 [details]
anaconda-yum.conf (with method=)
Comment 5 Kamil Páral 2013-11-05 09:05:47 EST
Created attachment 819779 [details]
packaging.log (with method=)
Comment 6 Kamil Páral 2013-11-05 09:05:55 EST
Created attachment 819780 [details]
program.log (with method=)
Comment 7 Kamil Páral 2013-11-05 09:06:03 EST
Created attachment 819781 [details]
storage.log (with method=)
Comment 8 Kamil Páral 2013-11-05 09:06:13 EST
Created attachment 819782 [details]
syslog (with method=)
Comment 9 Kamil Páral 2013-11-05 09:16:32 EST
Full anaconda logs available here:
http://kparal.fedorapeople.org/bugs/rhbz_1026834.7z
Comment 10 Kamil Páral 2013-11-28 08:39:33 EST
Created attachment 830241 [details]
screenshot with error in TC3

The reason why I've found this is because of bug 1026841. virt-install uses method= and therefore fails to install Fedora.

I have tried again with F20 TC3, the same problem occurs. This time it fails because of sqlite-3.8.1-2.fc20.x86_64 (which is in updates-testing, but not in stable repos). I used netinst and a kickstart. However, it's very interesting:

* BROKEN: method=http://dl.fedoraproject.org/pub/alt/stage/20-TC3/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

* WORKS: repo=http://dl.fedoraproject.org/pub/alt/stage/20-TC3/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

* WORKS: inst.repo=http://dl.fedoraproject.org/pub/alt/stage/20-TC3/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

* WORKS: method=http://dl.fedoraproject.org/pub/fedora/linux/development/20/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

* WORKS: repo=http://dl.fedoraproject.org/pub/fedora/linux/development/20/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

* WORKS: inst.repo=http://dl.fedoraproject.org/pub/fedora/linux/development/20/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

The only broken combination is method= _and_ a TC repository (i.e. not full development repository). The TC repository is just a small selection of packages (equivalent to loop-mounted DVD.iso), so I tried to replace it with a mounted DVD.iso image and it behaves the same - also spits out an error with method= (and not with repo= or inst.repo=).

I can't reproduce this with interactive installation, just with a kickstart.

The full TC3 logs are available at:
http://kparal.fedorapeople.org/bugs/rhbz_1026834_TC3.7z
Comment 11 Kamil Páral 2013-11-28 08:45:52 EST
This could be a violation of
"The installer must be able to complete a scripted installation which duplicates the default interactive installation as closely as possible. " or
"Any installation method or process designed to run unattended must do so. There should be no prompts requiring user intervention. "
https://fedoraproject.org/wiki/Fedora_20_Beta_Release_Criteria

Only method= + ks= installations seem to be affected, and only if you use DVD.iso-like repository. But it might be a common use case, if people are used to mount DVD.iso and then run "virt-install --location" (and a kickstart by any approach possible). That's exactly a combination of DVD repository, method= and ks=.
Comment 12 Petr Schindler 2013-11-28 08:47:57 EST
I reproduced failure from comment 10 with the same steps (kickstart and parameters).
Comment 13 Kamil Páral 2013-12-02 05:53:39 EST
In case I can't attend today's blocker bug meeting, I vote +1 blocker. We should either
a) make virt-install to issue repo= instead of method= (that's bug 1026841). If people use method= manually, it's their fault, it has been obsolete for ages.
or
b) make anaconda behave the same for method= and repo= (that could be a fairly simple fix, the code probably contains some switch that activates different code path for method= than for repo=). This would fix virt-install as well and no changes to it would be necessary.
Comment 14 Kamil Páral 2013-12-02 11:00:20 EST
I withdraw my previous blocker vote. I was convinced I can reproduce it any time, just by using an old DVD.iso or TC directory. But no, today neither me nor Martin Kolman can reproduce it, even when using a over old TC repository. It is clearly very much affected by current yum repos contents. So, probably -1 blocker right now until we can better demonstrate the affected use cases.

My latest suspicion is that this happens when the TC folder/DVD compose contains _newer_ packages than current stable repos. It was the case for ​sqlite-3.8.1-2.fc20 [1]. We will see when we have TC4 and it again includes some packages that are available in updates-testing and not in stable updates.

[1] https://fedorahosted.org/rel-eng/ticket/5808#comment:8
Comment 15 Adam Williamson 2013-12-02 12:40:47 EST
Discussed at 2013-12-02 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-12-02/f20-blocker-review-%234.2013-12-02-17.02.log.txt . Agreed to delay the decision until after this can be tested with TC4, as outlined in c#14.
Comment 16 Kamil Páral 2013-12-03 11:08:33 EST
Created attachment 832201 [details]
screenshot with error in TC4

Here we go with TC4 :-) 

error populating transaction: failure: Packages/f/fedora-release-20-1.noarch.rpm from anaconda: [Errno 256] No more mirrors to try.

These boot options were used:
method=http://dl.fedoraproject.org/pub/alt/stage/20-TC4/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

fedora-release-20-1.noarch.rpm is available at:
http://dl.fedoraproject.org/pub/alt/stage/20-TC4/Fedora/x86_64/os/Packages/f/
and also in updates-testing. But it's not available in stable updates (only older version).

If you perform a GNOME installation instead of a minimal one, it crashes on webkitgtk3-2.2.2-2.x86_64.fc20, available in TC repo, but unavailable in neither updates-testing nor stable updates.

We debugged this with mkolman and we found out that 'url' kickstart command clashes with 'method=' boot argument in these circumstances. If we remove 'url' from kickstart, everything works as expected (or if we use repo= instead of method=).

So, to recapitulate current findings:
1. This affects virt-install users that use --location with a DVD-sized repo (either mounted DVD.iso or a TC/RC compose repo) and a kickstart containing 'url' command. It affects them only in certain conditions, probably when their --location contains newer packages than what 'url' and default Fedora repos point at.
2. It can be worked around by
a) starting installation by hand with repo= (i.e. avoiding virt-install)
b) removing 'url' from kickstart

This bug lowers the reliability of our installation process, but no longer seems serious enough to warrant a release blocker, in my opinion.
Comment 17 Adam Williamson 2013-12-03 12:33:36 EST
IIUC, it's going to be just about impossible to hit post-release, right? As the remote repos will be at least as new as the installation media at release time, and only get 'newer' as time goes on.
Comment 18 Kamil Páral 2013-12-04 06:45:19 EST
(In reply to Adam Williamson from comment #17)
> IIUC, it's going to be just about impossible to hit post-release, right? As
> the remote repos will be at least as new as the installation media at
> release time, and only get 'newer' as time goes on.

Yes, currently it seems that way. I seem to hit it only when the provided repository in method= is newer than the default Fedora repositories, which should never happen after Fedora 20 is released. But I can't really vouch for that. It's a black box for me.
Comment 19 Kamil Páral 2013-12-04 07:10:59 EST
Hmm, let's complicate the things even more, shall we?

Today I tried to boot Final TC4 netinst with a very old TC repo:

method=http://dl.fedoraproject.org/pub/alt/stage/20-Alpha-TC3/Fedora/x86_64/os/ inst.ks=http://192.168.11.1:8000/ks.cfg

The kickstart contained:
> url --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-$releasever&arch=$basearch
> repo --name=fedora
> repo --name=updates
> repo --name=updates-testing
> 
> %packages --default
> %end

It failed to find liberation-fonts-common-1.07.3-2.fc20.noarch.rpm, which is available in both Alpha-TC3 repo:
http://dl.fedoraproject.org/pub/alt/stage/20-Alpha-TC3/Fedora/x86_64/os/Packages/l/
and current main repo:
http://dl.fedoraproject.org/pub/fedora/linux/development/20/x86_64/os/Packages/l/
(it's not available in updates-testing repo)

Full logs available at: http://kparal.fedorapeople.org/bugs/rhbz_1026834_liberation.7z


When I do the same with 20-Alpha-TC1 repo, it crashes because of ql2200-firmware-2.02.08-8.fc20.noarch.rpm. But when I repeat that with 20-Beta-TC3 repo, everything works OK.

Can somebody explain me _that_? This indicates that this might affect even a post-release installations, in certain (unknown conditions).
Comment 20 Kamil Páral 2013-12-04 08:39:38 EST
Additional information: If I replace 'url --mirrorlist=' in the kickstart with 'url --url=' command, it no longer crashes.
Comment 21 Kamil Páral 2013-12-04 08:41:22 EST
James, can you please give us some yum-related feedback? Thanks.
Comment 22 Mike Ruckman 2013-12-04 12:14:35 EST
Discussed in 2013-12-04 Blocker Review Meeting [1]. It was voted as a RejectedBlocker but an AcceptedFreezeException. Non-functional method affects only virt users and post-release users won't be probably affected. Rejected as blocker but it's consider as FE. 

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2013-12-04/
Comment 23 Kamil Páral 2013-12-05 10:01:54 EST
Fixing tags.
Comment 24 Tomáš Hozza 2013-12-05 10:14:09 EST
I'm experiencing the same issue when trying to install F20 TC4 in virt-manager.

In the GUI I select "Create a new VM" -> Network Install (HTTP, FTP, or NFS)...

Note that:
- I don't use any kickstart, but manually doing all of the installation
steps.
- I'm installing the minimal installation.
Comment 25 Martin Kolman 2013-12-05 13:33:14 EST
Fix has been posted for review.
Comment 26 Fedora Update System 2013-12-11 17:48:45 EST
anaconda-20.25.15-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/anaconda-20.25.15-1.fc20
Comment 27 Fedora Update System 2013-12-12 11:31:39 EST
Package anaconda-20.25.15-1.fc20, python-blivet-0.23.9-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-20.25.15-1.fc20 python-blivet-0.23.9-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23257/python-blivet-0.23.9-1.fc20,anaconda-20.25.15-1.fc20
then log in and leave karma (feedback).
Comment 28 Fedora Update System 2013-12-13 00:34:19 EST
anaconda-20.25.15-1.fc20, python-blivet-0.23.9-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.