Bug 1297762 - dnf failed to update through HTTP/1.1-compliant proxy
dnf failed to update through HTTP/1.1-compliant proxy
Product: Fedora
Classification: Fedora
Component: librepo (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: packaging-team-maint
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2016-01-12 07:23 EST by Alan Jenkins
Modified: 2016-11-24 10:29 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-11-24 10:29:05 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Alan Jenkins 2016-01-12 07:23:28 EST
I set up a package cache using the Polipo proxy (and Apache frontend to remap urls nicely).  It seemed to work, but later dnf failed to update.  I believe I can show there is a problem in dnf (or the mirror).

Version: dnf-1.1.5-1.fc23.noarch

The problem was the cache returned an old repodata.xml, pointing to files which no longer existed.

However Polipo is supposed to follow the HTTP specification, and I will show you where dnf (and yum) fall foul of the spec.

I wouldn't bother reproducing the exact failure I had.  It's sensitive to timing and maybe other things.  But it would be so simple to prevent, using a cache-buster when fetching repodata.xml.  

# dnf install dconf-editor
Error: Failed to synchronize cache for repo 'updates' from 'http://fedora-mirror.carrier.duckdns.org/pub/fedora/linux/updates/23/x86_64/': Yum repo downloading error: Downloading error(s): repodata/865e8db25f987620cf1885c87e34aba9a85c80832cbaa38111101046339ce939-prestodelta.xml.xz - Cannot download, all mirrors were already tried without success

# curl --head http://fedora-mirror/pub/fedora/linux/updates/23/x86_64/repodata/repomd.xml
HTTP/1.1 200 OK
Date: Tue, 12 Jan 2016 08:46:14 GMT
Server: Apache/2.4.7 (Ubuntu)
Content-Length: 4847
ETag: "12ef-5290cd296d540"
Last-Modified: Mon, 11 Jan 2016 10:58:53 GMT
Accept-Ranges: bytes
Content-Type: application/xml
Age: 7845

I verified the out-of-date repodata.xml was pointing to 865e8db2*.prestodelta.xml.xz.

For reasons I cannot find, requesting directly through Polipo busted the cache. Here is the up-to-date response:

# http_proxy=http://brick:8123 curl --head http://mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/updates/23/x86_64/repodata/repomd.xml
HTTP/1.1 200 OK
Content-Length: 4847
ETag: "12ef-5291df6fb2d80"
Date: Tue, 12 Jan 2016 10:57:23 GMT
Last-Modified: Tue, 12 Jan 2016 07:25:58 GMT
Server: Apache/2.4.7 (Ubuntu)
Accept-Ranges: bytes
Content-Type: application/xml
Connection: keep-alive

The problem is that without an explicit `Expires` header, HTTP caches get to use heuristics, to decide whether to revalidate or not.[1]

The solution for HTTP/1.1 is simple.  dnf should set `Cache-Control: max-age=0` when requesting repodata.xml.  This will force caches to revalidate repodata.xml on the "origin server".

The other alternative[2] is to fix the mirrors - they can send exactly the same header on responses to achieve the same effect.  It'd be very nice of them to respect HTTP and the REST architecture like this, but realistically dnf should be more tolerant of random mirrors.


[1] HTTP references:

* Point 3 under https://www.mnot.net/cache_docs/#WORK
* Latest spec. documents "Calculating Heuristic Freshness" at http://tools.ietf.org/html/rfc7234#section-4.2.2

[2] A third alternative would be to fix the mirroring protocol.  Keep previous versions of prestodelta.xml.xz etc. around for a time.  This would satisfy the usual HTTP heuristic, and allow the cache to generate a constant load on origin servers for *any* number of clients behind the cache.  Clients don't have to rely on the cache's heuristic either; they can set `Cache-Control: max-age=` to allow for the minimum expected interval between updates.    (I.e. expecting old files to be removed daily, set max-age to 12 hours).  But for that case you probably want a full mirror anyway, so it's  probably over-engineered.
Comment 1 Alan Jenkins 2016-01-12 07:45:05 EST
Change component to librepo.  The function responsible is lr_yum_download_repomd().

(In turn, it calls lr_downloadtarget_new(), which has no options for caching / HTTP header override).

Additional note: Yum didn't use Cache-Control either, so AFAICT this bug isn't a regression.
Comment 2 Alan Jenkins 2016-01-12 11:55:54 EST
Patch: https://github.com/rpm-software-management/librepo/pull/77
Comment 3 Tomas Mlcoch 2016-03-04 03:37:18 EST
Fixed in upstream in commits:

Comment 4 Fedora End Of Life 2016-11-24 09:52:17 EST
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Note You need to log in before you can comment on or make changes to this bug.