Bug 1637148 - dnf doesn't resolve variables in mirrorlists
Summary: dnf doesn't resolve variables in mirrorlists
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 29
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Marek Blaha
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeExcepti...
: 1645413 1646894 (view as bug list)
Depends On:
Blocks: F29FinalFreezeException 1651092
TreeView+ depends on / blocked
 
Reported: 2018-10-08 18:38 UTC by Michael
Modified: 2019-05-28 02:33 UTC (History)
17 users (show)

Fixed In Version: dnf-4.0.9-1.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1651092 (view as bug list)
Environment:
Last Closed: 2018-12-05 02:34:18 UTC
Type: Bug


Attachments (Terms of Use)

Description Michael 2018-10-08 18:38:09 UTC
Description of problem:
Some yum/dnf repositories contain links to online mirrorlists. Those mirrorlists may use variables like $releasever or $basearch within its URLs. Until Fedora 28 this worked flawlessly. With Fedora 29 beta the DNF variables are NOT resolved any more.

Version-Release number of selected component (if applicable):
DNF version: 3.6.1

How reproducible:
1. Install a repository which uses an online mirrorlist (in my case it's unitedrpms).
2. sudo dnf --disablerepo '*' --enablerepo unitedrpms upgrade --refresh --verbose

Actual results:
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync, system-upgrade
DNF version: 3.6.1
cachedir: /var/cache/dnf
repo: downloading from remote: unitedrpms
error: Status code: 404 for https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml (https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml).
error: Status code: 404 for https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/ (https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/repodata/repomd.xml/).
unitedrpms 29 - x86_64   
Cannot download 'https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
Failed to synchronize cache for repo 'unitedrpms', ignoring this repo.
--> Starting dependency resolution
--> Finished dependency resolution
Dependencies resolved.
Nothing to do.
Complete!

Expected results:
DNF variables within URLs of mirrors shall be resolved/replaced.

e.g.: https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/repodata/repomd.xml shall be transformed into https://sourceforge.net/projects/unitedrpms/files/29/x86_64/repodata/repomd.xml

Comment 1 Michael 2018-10-09 09:22:11 UTC
This is the mirrorlist whose URLs are not treated properly by DNF in my case:
https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

Content:
https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

My guess is that many repositories which use that style of mirrorlist are affected.

Comment 2 Kamil Páral 2018-10-09 11:07:34 UTC
I can't reproduce this. If I take fedora repos and use baseurl (including both $releasever and $basearch) instead of mirrorlist, dnf still resolves that ok and pulls the metadata.
dnf-3.6.1-1.fc29.noarch

Michael, please upload the repo file that doesn't work for you.

Comment 3 Michael 2018-10-09 11:16:24 UTC
Somebody implemented a workaround (!) on UnitedRPMs side already (see https://github.com/UnitedRPMs/unitedrpms/commit/30c505cadaf0145986f7eae938573a32ba7a6316).

However, that doesn't fix the root cause of the issue...

Here's the (original) repo file I was writing about:

/etc/yum.repos.d/unitedrpms.repo:
---------------------------------------------
[unitedrpms]
name=unitedrpms $releasever - $basearch
mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt
baseurl=https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
enabled=1
priority=1
metadata_expire=1d
skip_if_unavailable=1
gpgkey=file:///etc/pki/rpm-gpg/URPMS-GPG-PUBLICKEY-Fedora-24
gpgcheck=1
repo_gpgcheck=1
enabled_metadata=1
---------------------------------------------

Comment 4 Michael 2018-10-09 11:23:46 UTC
The original state of the repository file, I mentioned, can also be obtained by using the previous version: https://github.com/UnitedRPMs/unitedrpms/releases/download/8/unitedrpms-29-8.fc29.noarch.rpm

Comment 5 Kamil Páral 2018-10-09 11:43:42 UTC
OK, also I got confused by metalink vs mirrorlist. Nevertheless, in a repo with this:

mirrorlist=https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt

I can reproduce the problem. The hyperlink returns this content:

https://sourceforge.net/projects/unitedrpms/files/$releasever/$basearch/
https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

at that's later used literally. So it seems that the variables are replaced in baseurl/metalink/mirrorlist values, but not in mirrorlist _content_. I don't know whether that's expected or not (as you say, it worked before). Now we need feedback from DNF devs.

Comment 6 Michael 2018-10-09 11:51:42 UTC
Exactly. :-)
The content (URLs of the mirrors) inside the mirrorlist file isn't treated right in my opinion.

I'm very certain that it worked fine the last time I used fedora 28.

Comment 7 Neal Gompa 2018-10-09 12:55:56 UTC
I'm not sure this was supposed to work. My understanding is that the mirrorlist text file itself was supposed to be variable substituted, and it would pass in a literal newline separated list of _valid_ URLs.

If this worked before, I'm not sure it was supposed to...

Comment 8 Kamil Páral 2018-10-09 13:03:17 UTC
(In reply to Michael from comment #6)
> I'm very certain that it worked fine the last time I used fedora 28.

Confirmed, it works fine with dnf-2.7.5-12.fc28. That doesn't mean it was a documented supported behavior, of course.

Comment 9 Neal Gompa 2018-10-09 15:12:06 UTC
(In reply to Neal Gompa from comment #7)
> I'm not sure this was supposed to work. My understanding is that the
> mirrorlist text file itself was supposed to be variable substituted, and it
> would pass in a literal newline separated list of _valid_ URLs.
> 
> If this worked before, I'm not sure it was supposed to...

Sorry, I meant mirrorlist text file URL path, not the file content.

Comment 11 Geoffrey Marr 2018-10-15 19:12:51 UTC
Discussed during the 2018-10-15 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as this is not believed to affect any official / out-of-the-box repositories and thus does not break any release criteria. As this could benefit some folks on first boot, it is accepted as a freeze exception if the fix is simple and testable.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-15/f29-blocker-review.2018-10-15-16.00.txt

Comment 12 Marek Blaha 2018-11-05 07:53:51 UTC
*** Bug 1645413 has been marked as a duplicate of this bug. ***

Comment 13 Marek Blaha 2018-11-07 06:21:11 UTC
*** Bug 1646894 has been marked as a duplicate of this bug. ***

Comment 14 Fedora Update System 2018-11-22 18:57:23 UTC
libdnf-0.22.3-1.fc29 dnf-4.0.9-1.fc29 dnf-plugins-core-4.0.2-1.fc29 dnf-plugins-extras-4.0.0-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-17cbc3c616

Comment 15 Fedora Update System 2018-11-23 02:56:56 UTC
dnf-4.0.9-1.fc29, dnf-plugins-core-4.0.2-1.fc29, dnf-plugins-extras-4.0.0-1.fc29, libdnf-0.22.3-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-17cbc3c616

Comment 16 Fedora Update System 2018-12-05 02:34:18 UTC
dnf-4.0.9-1.fc29, dnf-plugins-core-4.0.2-1.fc29, dnf-plugins-extras-4.0.0-1.fc29, libdnf-0.22.3-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 RobbieTheK 2019-05-28 02:33:08 UTC
On Fedora 29 I'm seeing Failed to synchronize cache for repo 'unitedrpms'

I have:
mirrorlist=https://sourceforge.net/projects/unitedrpms/files/29/x86_64/
baseurl=https://osdn.net/projects/unitedrpms/storage/29/x86_64/

Based on: https://github.com/UnitedRPMs/unitedrpms/commit/30c505cadaf0145986f7eae938573a32ba7a6316


Note You need to log in before you can comment on or make changes to this bug.