Bug 968006

Summary: [rfe] download filelists only on demand
Product: [Fedora] Fedora Reporter: Rahul Sundaram <metherid>
Component: dnfAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED DEFERRED QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: akozumpl, fweimer, hedayatv, jsilhan, jzeleny, mls, packaging-team-maint, pmatilai, samuel-rhbugs, tuksgig, vondruch
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-23 11:55:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Rahul Sundaram 2013-05-28 18:48:05 UTC
Description of problem:

Unlike yum, dnf seems to download the full filelists always and hence the initial metadata download is several dozen MB's larger and the user has to pay the cost of keeping it in sync as well but the full filelist is rarely necessary since some of the common paths including /usr/bin etc is already covered by the other parts of metadata




Comment 1 Ales Kozumplik 2013-05-29 07:27:50 UTC
yes, this is a problem but I am not sure what can be done about it. The metadata are just poorly designed. 

An alternative worth considering is simply failing to satisfy deps of all packages that depend on files outside /usr/bin etc.

Comment 2 Rahul Sundaram 2013-05-29 07:31:12 UTC
Do what yum does for now.  don't download the entire filelist always but on demand when someone runs say dnf install /path/to/foo if that path is outside the usual ones

Comment 3 Ales Kozumplik 2013-05-29 07:40:53 UTC
Some packages have e.g.

Requires: /var/something/weird

And whenever we need to install one of these we need the full filelists. It's impossible to tell ahead of resolving what packages will end up in the transaction so we always need the filelists now.

One alternative is opting-in to a libsolv callback that asks for the full filelists as soon as we encounter first such package. The downsides of this are:

:: conflicts the current DNF strategy to pre-sync all metadata in background
:: confusing DNF operation: user sees metadata sync, then resolving, then some other metadata sync.
:: code complexity including callbacks across many layers.

I think the right way is to fix packaging policy and fail to build any packages that Require: anything outside the few correct paths.

Comment 4 Panu Matilainen 2013-05-29 07:56:01 UTC
It's nowhere as simple as "do what yum does", because what yum does is largely tied to the "lazy" way how yum's depsolver works by looking up each dep as it comes across them, and the metadata layout is heavily influenced by how yum works. The metadata format is pretty horrible for EVERYTHING else, not just dnf (ie libsolv) but smartpm, apt-rpm etc etc too which look at the dependency set as whole, and thus require that either all dependencies are satisfiable within the primary metadata or that filelists are always downloaded.

Comment 5 Rahul Sundaram 2013-05-29 08:15:47 UTC
If this is a problem that needs to be fixed by better packaging,  I think two things would be needed. 

a) a list of packages that use the uncommon paths to fedora devel list so that we can get them fixed

b) suggested tweak to the packaging guideline needs to be filed with fpc

Then dnf can stop downloading the filelists entirely

Comment 6 Panu Matilainen 2013-05-29 08:20:30 UTC
No it can't, 3rd party repositories and packages are not forced to follow fedora policies.

Comment 7 Rahul Sundaram 2013-05-29 08:37:56 UTC
They aren't forced to but common third party repos for Fedora generally do follow the Fedora packaging guidelines (other than licensing) and file based dependencies are fairly unusual in them anyway.

Comment 8 Jan Zeleny 2013-05-29 10:46:59 UTC
I don't think you can generalize like that. There might be some repositories that are using this feature and you are just aware of them. Cutting the support without any warning, *discussion* and transition period would be just wrong and insensitive.

Comment 9 Ales Kozumplik 2013-05-29 10:51:33 UTC
All in all, we neither have a good way, will or resources to pull this off. Closing.

Comment 10 Rahul Sundaram 2013-05-29 17:05:02 UTC
@Jan Zeleny, I completely agree and that is specifically why I was outlining both a discussion in devel list and a transitioning period by drafting a packaging guideline update and then doing a change but since there is no will to push through for a change, I will let it be.  It remains a large problem for people on slow net connections however. Yum itself wasn't all that great and dnf is making it worse by pulling in the filelists all the time.

Comment 11 Panu Matilainen 2013-05-30 05:20:55 UTC
Rahul, if you want to drive such a Fedora policy change then go ahead and make the initiative, we're not going to stop you but we're not into Fedora politics either. Such a change would help yum *now*, which is still the standard depsolver for the time being afterall, and what dnf does at the time its time to switch over ... well, there's plenty of time to come up with something more clever, such as repodata extensions to limit the filelists "damage".

Comment 12 Rahul Sundaram 2013-05-30 05:34:16 UTC
I suspect noone here has ever used yum on a dial-up connection.  This isn't about Fedora politics as much as helping low bandwidth users (of which I was one for years). I wouldn't have the time to work on this all by myself but I have posted a few ideas to devel list.

Comment 13 Panu Matilainen 2013-05-30 05:48:12 UTC
Whether abritrary file dependencies are permitted in Fedora or not, when the underlying package format allows them, is nothing but Fedora politics and not a technical issue. If you want to see such a change, I suggest you drive it instead of trying to find somebody else to do it, the packaging team is busy as anybody (with the actual technical work).

Nobody disagrees the metadata is bloated and hideous for slow connections, but that problem is not something that can be solved within dnf alone.

Comment 14 Rahul Sundaram 2013-05-30 06:50:27 UTC
It requires packaging changes across several packages and wouldn't be feasible for me to drive entirely as my volunteer time is limited but I have posted some ideas and if there is some consensus,  I will file a ticket with FESCo or FPC.  Thanks!

Comment 15 Michael Schröder 2013-06-03 10:00:31 UTC
(Just for clarification: As Ales mentioned, libsolv *has* a callback for lazy downloading of metadata, so it can work pretty much the same way as yum did.

So it's more a dnf design decision.)

Comment 16 Hedayat Vatankhah 2014-06-14 01:33:37 UTC
Would you please re-consider marking this bug as CANTFIX? I try dnf from time to time and I was going to fill exactly this bug report. Yum downloads about 12MB metadata for updates repository (which is already a waste of bandwidth, and too much for some internet users), but DNF downloads around 30MB, which is horrible! In my Fedora 20, yum has NEVER downloaded file lists for updates repository. 

Yum has always been one of the parts of Fedora that I disliked very much because of wasting BW on downloading metadata, and DNF has made it even worse! I wonder how DNF is going to replace YUM with this problem.

Comment 17 Ales Kozumplik 2014-06-23 11:55:24 UTC
As described above, there is nothing we can do about this in the foreseeable future, sorry.

Comment 18 Hedayat Vatankhah 2014-06-23 12:12:27 UTC
OK. As far as I can see (according to comment #15) it is possible. But if you don't want, it is another issue.

Comment 19 Honza Silhan 2015-03-04 13:37:38 UTC
If anyone find the new use case other than executing the scriplets again, don't hesitate to share it and we will reconsidered it.

Comment 20 Honza Silhan 2015-03-04 13:41:24 UTC
* sry, the comment was to other bug report but this could be reconsidered as well if bug 850896 doesn't fix it.

Comment 21 Radek Holy 2015-07-03 12:17:01 UTC
*** Bug 1239066 has been marked as a duplicate of this bug. ***