Bug 1637148 - dnf doesn't resolve variables in mirrorlists
Summary: dnf doesn't resolve variables in mirrorlists
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 29
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Marek Blaha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeExcepti...
Keywords: CommonBugs, Triaged
: 1645413 1646894 (view as bug list)
Depends On:
Blocks: 1651092 F29FinalFreezeException
TreeView+ depends on / blocked
 
Reported: 2018-10-08 18:38 UTC by Michael
Modified: 2018-12-05 02:34 UTC (History)
16 users (show)

(edit)
Clone Of:
: 1651092 (view as bug list)
(edit)
Last Closed: 2018-12-05 02:34:18 UTC


Attachments (Terms of Use)

Description Michael 2018-10-08 18:38:09 UTC
Description of problem:
Some yum/dnf repositories contain links to online mirrorlists. Those mirrorlists may use variables like $releasever or $basearch within its URLs. Until Fedora 28 this worked flawlessly. With Fedora 29 beta the DNF variables are NOT resolved any more.

Version-Release number of selected component (if applicable):
DNF version: 3.6.1

How reproducible:
1. Install a repository which uses an online mirrorlist (in my case it's unitedrpms).
2. sudo dnf --disablerepo '*' --enablerepo unitedrpms upgrade --refresh --verbose

Actual results:
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync, system-upgrade
DNF version: 3.6.1
cachedir: /var/cache/dnf
repo: downloading from remote: unitedrpms
error: Status code: 404 for https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml (https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml).
error: Status code: 404 for https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/ (https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/).
unitedrpms 29 - x86_64   
Cannot download 'https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
Failed to synchronize cache for repo 'unitedrpms', ignoring this repo.
--> Starting dependency resolution
--> Finished dependency resolution
Dependencies resolved.
Nothing to do.
Complete!

Expected results:
DNF variables within URLs of mirrors shall be resolved/replaced.

e.g.: https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml shall be transformed into https://sourceforge.net/projects/unitedrpms/files/29/x86_64/repodata/repomd.xml

Comment 1 Michael 2018-10-09 09:22:11 UTC
This is the mirrorlist whose URLs are not treated properly by DNF in my case:
https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

Content:
https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

My guess is that many repositories which use that style of mirrorlist are affected.

Comment 2 Kamil Páral 2018-10-09 11:07:34 UTC
I can't reproduce this. If I take fedora repos and use baseurl (including both $releasever and $basearch) instead of mirrorlist, dnf still resolves that ok and pulls the metadata.
dnf-3.6.1-1.fc29.noarch

Michael, please upload the repo file that doesn't work for you.

Comment 3 Michael 2018-10-09 11:16:24 UTC
Somebody implemented a workaround (!) on UnitedRPMs side already (see https://github.com/UnitedRPMs/unitedrpms/commit/30c505cadaf0145986f7eae938573a32ba7a6316).

However, that doesn't fix the root cause of the issue...

Here's the (original) repo file I was writing about:

/etc/yum.repos.d/unitedrpms.repo:
---------------------------------------------
[unitedrpms]
name=unitedrpms $releasever - $basearch
mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt
baseurl=https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
enabled=1
priority=1
metadata_expire=1d
skip_if_unavailable=1
gpgkey=file:///etc/pki/rpm-gpg/URPMS-GPG-PUBLICKEY-Fedora-24
gpgcheck=1
repo_gpgcheck=1
enabled_metadata=1
---------------------------------------------

Comment 4 Michael 2018-10-09 11:23:46 UTC
The original state of the repository file, I mentioned, can also be obtained by using the previous version: https://github.com/UnitedRPMs/unitedrpms/releases/download/8/unitedrpms-29-8.fc29.noarch.rpm

Comment 5 Kamil Páral 2018-10-09 11:43:42 UTC
OK, also I got confused by metalink vs mirrorlist. Nevertheless, in a repo with this:

mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

I can reproduce the problem. The hyperlink returns this content:

https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

at that's later used literally. So it seems that the variables are replaced in baseurl/metalink/mirrorlist values, but not in mirrorlist _content_. I don't know whether that's expected or not (as you say, it worked before). Now we need feedback from DNF devs.

Comment 6 Michael 2018-10-09 11:51:42 UTC
Exactly. :-)
The content (URLs of the mirrors) inside the mirrorlist file isn't treated right in my opinion.

I'm very certain that it worked fine the last time I used fedora 28.

Comment 7 Neal Gompa 2018-10-09 12:55:56 UTC
I'm not sure this was supposed to work. My understanding is that the mirrorlist text file itself was supposed to be variable substituted, and it would pass in a literal newline separated list of _valid_ URLs.

If this worked before, I'm not sure it was supposed to...

Comment 8 Kamil Páral 2018-10-09 13:03:17 UTC
(In reply to Michael from comment #6)
> I'm very certain that it worked fine the last time I used fedora 28.

Confirmed, it works fine with dnf-2.7.5-12.fc28. That doesn't mean it was a documented supported behavior, of course.

Comment 9 Neal Gompa 2018-10-09 15:12:06 UTC
(In reply to Neal Gompa from comment #7)
> I'm not sure this was supposed to work. My understanding is that the
> mirrorlist text file itself was supposed to be variable substituted, and it
> would pass in a literal newline separated list of _valid_ URLs.
> 
> If this worked before, I'm not sure it was supposed to...

Sorry, I meant mirrorlist text file URL path, not the file content.

Comment 11 Geoffrey Marr 2018-10-15 19:12:51 UTC
Discussed during the 2018-10-15 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as this is not believed to affect any official / out-of-the-box repositories and thus does not break any release criteria. As this could benefit some folks on first boot, it is accepted as a freeze exception if the fix is simple and testable.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-15/f29-blocker-review.2018-10-15-16.00.txt

Comment 12 Marek Blaha 2018-11-05 07:53:51 UTC
*** Bug 1645413 has been marked as a duplicate of this bug. ***

Comment 13 Marek Blaha 2018-11-07 06:21:11 UTC
*** Bug 1646894 has been marked as a duplicate of this bug. ***

Comment 14 Fedora Update System 2018-11-22 18:57:23 UTC
libdnf-0.22.3-1.fc29 dnf-4.0.9-1.fc29 dnf-plugins-core-4.0.2-1.fc29 dnf-plugins-extras-4.0.0-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-17cbc3c616

Comment 15 Fedora Update System 2018-11-23 02:56:56 UTC
dnf-4.0.9-1.fc29, dnf-plugins-core-4.0.2-1.fc29, dnf-plugins-extras-4.0.0-1.fc29, libdnf-0.22.3-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-17cbc3c616

Comment 16 Fedora Update System 2018-12-05 02:34:18 UTC
dnf-4.0.9-1.fc29, dnf-plugins-core-4.0.2-1.fc29, dnf-plugins-extras-4.0.0-1.fc29, libdnf-0.22.3-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.