Red Hat Bugzilla – Bug 968006
[rfe] download filelists only on demand
Last modified: 2015-11-25 01:43:11 EST
Description of problem:
Unlike yum, dnf seems to download the full filelists always and hence the initial metadata download is several dozen MB's larger and the user has to pay the cost of keeping it in sync as well but the full filelist is rarely necessary since some of the common paths including /usr/bin etc is already covered by the other parts of metadata
yes, this is a problem but I am not sure what can be done about it. The metadata are just poorly designed.
An alternative worth considering is simply failing to satisfy deps of all packages that depend on files outside /usr/bin etc.
Do what yum does for now. don't download the entire filelist always but on demand when someone runs say dnf install /path/to/foo if that path is outside the usual ones
Some packages have e.g.
And whenever we need to install one of these we need the full filelists. It's impossible to tell ahead of resolving what packages will end up in the transaction so we always need the filelists now.
One alternative is opting-in to a libsolv callback that asks for the full filelists as soon as we encounter first such package. The downsides of this are:
:: conflicts the current DNF strategy to pre-sync all metadata in background
:: confusing DNF operation: user sees metadata sync, then resolving, then some other metadata sync.
:: code complexity including callbacks across many layers.
I think the right way is to fix packaging policy and fail to build any packages that Require: anything outside the few correct paths.
It's nowhere as simple as "do what yum does", because what yum does is largely tied to the "lazy" way how yum's depsolver works by looking up each dep as it comes across them, and the metadata layout is heavily influenced by how yum works. The metadata format is pretty horrible for EVERYTHING else, not just dnf (ie libsolv) but smartpm, apt-rpm etc etc too which look at the dependency set as whole, and thus require that either all dependencies are satisfiable within the primary metadata or that filelists are always downloaded.
If this is a problem that needs to be fixed by better packaging, I think two things would be needed.
a) a list of packages that use the uncommon paths to fedora devel list so that we can get them fixed
b) suggested tweak to the packaging guideline needs to be filed with fpc
Then dnf can stop downloading the filelists entirely
No it can't, 3rd party repositories and packages are not forced to follow fedora policies.
They aren't forced to but common third party repos for Fedora generally do follow the Fedora packaging guidelines (other than licensing) and file based dependencies are fairly unusual in them anyway.
I don't think you can generalize like that. There might be some repositories that are using this feature and you are just aware of them. Cutting the support without any warning, *discussion* and transition period would be just wrong and insensitive.
All in all, we neither have a good way, will or resources to pull this off. Closing.
@Jan Zeleny, I completely agree and that is specifically why I was outlining both a discussion in devel list and a transitioning period by drafting a packaging guideline update and then doing a change but since there is no will to push through for a change, I will let it be. It remains a large problem for people on slow net connections however. Yum itself wasn't all that great and dnf is making it worse by pulling in the filelists all the time.
Rahul, if you want to drive such a Fedora policy change then go ahead and make the initiative, we're not going to stop you but we're not into Fedora politics either. Such a change would help yum *now*, which is still the standard depsolver for the time being afterall, and what dnf does at the time its time to switch over ... well, there's plenty of time to come up with something more clever, such as repodata extensions to limit the filelists "damage".
I suspect noone here has ever used yum on a dial-up connection. This isn't about Fedora politics as much as helping low bandwidth users (of which I was one for years). I wouldn't have the time to work on this all by myself but I have posted a few ideas to devel list.
Whether abritrary file dependencies are permitted in Fedora or not, when the underlying package format allows them, is nothing but Fedora politics and not a technical issue. If you want to see such a change, I suggest you drive it instead of trying to find somebody else to do it, the packaging team is busy as anybody (with the actual technical work).
Nobody disagrees the metadata is bloated and hideous for slow connections, but that problem is not something that can be solved within dnf alone.
It requires packaging changes across several packages and wouldn't be feasible for me to drive entirely as my volunteer time is limited but I have posted some ideas and if there is some consensus, I will file a ticket with FESCo or FPC. Thanks!
(Just for clarification: As Ales mentioned, libsolv *has* a callback for lazy downloading of metadata, so it can work pretty much the same way as yum did.
So it's more a dnf design decision.)
Would you please re-consider marking this bug as CANTFIX? I try dnf from time to time and I was going to fill exactly this bug report. Yum downloads about 12MB metadata for updates repository (which is already a waste of bandwidth, and too much for some internet users), but DNF downloads around 30MB, which is horrible! In my Fedora 20, yum has NEVER downloaded file lists for updates repository.
Yum has always been one of the parts of Fedora that I disliked very much because of wasting BW on downloading metadata, and DNF has made it even worse! I wonder how DNF is going to replace YUM with this problem.
As described above, there is nothing we can do about this in the foreseeable future, sorry.
OK. As far as I can see (according to comment #15) it is possible. But if you don't want, it is another issue.
If anyone find the new use case other than executing the scriplets again, don't hesitate to share it and we will reconsidered it.
* sry, the comment was to other bug report but this could be reconsidered as well if bug 850896 doesn't fix it.
*** Bug 1239066 has been marked as a duplicate of this bug. ***