Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1637148 - dnf doesn't resolve variables in mirrorlists
dnf doesn't resolve variables in mirrorlists
Status: POST
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
29
All Linux
unspecified Severity high
: ---
: ---
Assigned To: Marek Blaha
Fedora Extras Quality Assurance
RejectedBlocker AcceptedFreezeExcepti...
: CommonBugs, Triaged
: 1645413 1646894 (view as bug list)
Depends On:
Blocks: F29FinalFreezeException
  Show dependency treegraph
 
Reported: 2018-10-08 14:38 EDT by Michael
Modified: 2018-11-07 01:44 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Michael 2018-10-08 14:38:09 EDT
Description of problem:
Some yum/dnf repositories contain links to online mirrorlists. Those mirrorlists may use variables like $releasever or $basearch within its URLs. Until Fedora 28 this worked flawlessly. With Fedora 29 beta the DNF variables are NOT resolved any more.

Version-Release number of selected component (if applicable):
DNF version: 3.6.1

How reproducible:
1. Install a repository which uses an online mirrorlist (in my case it's unitedrpms).
2. sudo dnf --disablerepo '*' --enablerepo unitedrpms upgrade --refresh --verbose

Actual results:
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync, system-upgrade
DNF version: 3.6.1
cachedir: /var/cache/dnf
repo: downloading from remote: unitedrpms
error: Status code: 404 for https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml (https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml).
error: Status code: 404 for https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/ (https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/).
unitedrpms 29 - x86_64   
Cannot download 'https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
Failed to synchronize cache for repo 'unitedrpms', ignoring this repo.
--> Starting dependency resolution
--> Finished dependency resolution
Dependencies resolved.
Nothing to do.
Complete!

Expected results:
DNF variables within URLs of mirrors shall be resolved/replaced.

e.g.: https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml shall be transformed into https://sourceforge.net/projects/unitedrpms/files/29/x86_64/repodata/repomd.xml
Comment 1 Michael 2018-10-09 05:22:11 EDT
This is the mirrorlist whose URLs are not treated properly by DNF in my case:
https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

Content:
https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

My guess is that many repositories which use that style of mirrorlist are affected.
Comment 2 Kamil Páral 2018-10-09 07:07:34 EDT
I can't reproduce this. If I take fedora repos and use baseurl (including both $releasever and $basearch) instead of mirrorlist, dnf still resolves that ok and pulls the metadata.
dnf-3.6.1-1.fc29.noarch

Michael, please upload the repo file that doesn't work for you.
Comment 3 Michael 2018-10-09 07:16:24 EDT
Somebody implemented a workaround (!) on UnitedRPMs side already (see https://github.com/UnitedRPMs/unitedrpms/commit/30c505cadaf0145986f7eae938573a32ba7a6316).

However, that doesn't fix the root cause of the issue...

Here's the (original) repo file I was writing about:

/etc/yum.repos.d/unitedrpms.repo:
---------------------------------------------
[unitedrpms]
name=unitedrpms $releasever - $basearch
mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt
baseurl=https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
enabled=1
priority=1
metadata_expire=1d
skip_if_unavailable=1
gpgkey=file:///etc/pki/rpm-gpg/URPMS-GPG-PUBLICKEY-Fedora-24
gpgcheck=1
repo_gpgcheck=1
enabled_metadata=1
---------------------------------------------
Comment 4 Michael 2018-10-09 07:23:46 EDT
The original state of the repository file, I mentioned, can also be obtained by using the previous version: https://github.com/UnitedRPMs/unitedrpms/releases/download/8/unitedrpms-29-8.fc29.noarch.rpm
Comment 5 Kamil Páral 2018-10-09 07:43:42 EDT
OK, also I got confused by metalink vs mirrorlist. Nevertheless, in a repo with this:

mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

I can reproduce the problem. The hyperlink returns this content:

https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

at that's later used literally. So it seems that the variables are replaced in baseurl/metalink/mirrorlist values, but not in mirrorlist _content_. I don't know whether that's expected or not (as you say, it worked before). Now we need feedback from DNF devs.
Comment 6 Michael 2018-10-09 07:51:42 EDT
Exactly. :-)
The content (URLs of the mirrors) inside the mirrorlist file isn't treated right in my opinion.

I'm very certain that it worked fine the last time I used fedora 28.
Comment 7 Neal Gompa 2018-10-09 08:55:56 EDT
I'm not sure this was supposed to work. My understanding is that the mirrorlist text file itself was supposed to be variable substituted, and it would pass in a literal newline separated list of _valid_ URLs.

If this worked before, I'm not sure it was supposed to...
Comment 8 Kamil Páral 2018-10-09 09:03:17 EDT
(In reply to Michael from comment #6)
> I'm very certain that it worked fine the last time I used fedora 28.

Confirmed, it works fine with dnf-2.7.5-12.fc28. That doesn't mean it was a documented supported behavior, of course.
Comment 9 Neal Gompa 2018-10-09 11:12:06 EDT
(In reply to Neal Gompa from comment #7)
> I'm not sure this was supposed to work. My understanding is that the
> mirrorlist text file itself was supposed to be variable substituted, and it
> would pass in a literal newline separated list of _valid_ URLs.
> 
> If this worked before, I'm not sure it was supposed to...

Sorry, I meant mirrorlist text file URL path, not the file content.
Comment 11 Geoffrey Marr 2018-10-15 15:12:51 EDT
Discussed during the 2018-10-15 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as this is not believed to affect any official / out-of-the-box repositories and thus does not break any release criteria. As this could benefit some folks on first boot, it is accepted as a freeze exception if the fix is simple and testable.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-15/f29-blocker-review.2018-10-15-16.00.txt
Comment 12 Marek Blaha 2018-11-05 02:53:51 EST
*** Bug 1645413 has been marked as a duplicate of this bug. ***
Comment 13 Marek Blaha 2018-11-07 01:21:11 EST
*** Bug 1646894 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.