Bug 1816153 - [RFE] Redownload repomd.xml if downloaded repomd.xml points to non-existing files
Summary: [RFE] Redownload repomd.xml if downloaded repomd.xml points to non-existing f...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1836518 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-23 13:10 UTC by Adrian Reber
Modified: 2023-08-28 11:54 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adrian Reber 2020-03-23 13:10:35 UTC
This is complicated and based on https://pagure.io/fedora-infrastructure/issue/8679

The problem is currently realated to COPR in AWS using S3 as mirror via cloudfront. If I understood it correctly.

Fedora started to sync its master mirror to S3 to be available faster and cheaper in AWS. Although we believe we are doing everything correctly (see pagure ticket) COPR still sees failures.

The current steps are:

 * sync everything to S3 besides repodata
 * sync only repodata
 * invalidate repomd.xml in cloudfront
 * sync with --delete

Unfortunately there are still problems which are related to cloudfront caching (that's what we currently think at least).

Although we are telling cloudfront to not cache repomd.xml it seems that sometimes COPR clients are getting repomd.xml files (which are still referenced and valid in the metalink) which are pointing to repodata files which are not available on any mirror any more.

So this RFE would like to propose that DNF tries to redownload another repomd.xml, with another checksum from the metalink, if all files mentioned in repomd.xml are not available on any mirror.

So instead of:

 * get metalink
 * get corresponding repomd.xml from first mirror (not sure DNF uses the first mirror to download repomd.xml)
 * check if repomd.xml matches one of the checksums
 * download files mentioned in the repomd.xml file

If the last step fails, as it can be seen in the pagure ticket, mark the repomd.xml file with the corresponding checksum just downloaded as invalid and try again with the remaining repomd.xml checksums mentioned in the metalink.

This way a partially synced mirror (or in the middle of a sync) or a wrongly cached file (like cloudfront) would not abort the complete DNF transaction.

Another way to make it more robust would be to first try to download the newest repomd.xml file mentioned in the metalink and go to the next mirror until the newest repomd.xml file was downloaded. Only if no mirror was able to provide the newest repomd.xml file, try the alternative repomd.xml files mentioned in the metalink.

I did not look at the DNF code, so please correct me if I am assuming anything which is not true.

Comment 1 Lukáš Hrázký 2020-06-01 12:41:26 UTC
*** Bug 1836518 has been marked as a duplicate of this bug. ***

Comment 2 Fedora Program Management 2021-04-29 16:15:15 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.


Note You need to log in before you can comment on or make changes to this bug.