Description of problem: DNF downloads uncompressed comps.xml in case the gzipped variant is not available. xz or zck variants are not used at all. It is wasting of network transfer capacity - plain xml is large and DNF does not utilize the benefits of zchunk (well, this is questionable given that comps.xml is not updated that much during a release cycle). How reproducible: 100% Steps to Reproduce: 1. have a repository without group_gz metadata (e.g. fedora repository from F32 - no group_gz metadata available) Here is excerpt from F32 repomd.xml containing group metadata: ---- 8< ---- <data type="group"> <checksum type="sha256">961f955de96f1489e49012cfba125d771d2b4dce6cb31678aea644dffcfb3935</checksum> <location href="repodata/961f955de96f1489e49012cfba125d771d2b4dce6cb31678aea644dffcfb3935-comps-Everything.x86_64.xml"/> <timestamp>1587588985</timestamp> <size>1772199</size> </data> <data type="group_xz"> <checksum type="sha256">e07100d0ba13f87849cd17d7b9e2a2a6033310983042c8cf5461141009436373</checksum> <location href="repodata/e07100d0ba13f87849cd17d7b9e2a2a6033310983042c8cf5461141009436373-comps-Everything.x86_64.xml.xz"/> <timestamp>1587594018</timestamp> <size>259956</size> </data> <data type="group_zck"> <checksum type="sha256">b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47</checksum> <open-checksum type="sha256">e07100d0ba13f87849cd17d7b9e2a2a6033310983042c8cf5461141009436373</open-checksum> <header-checksum type="sha256">70457131f086c5ecd94393551d4d1ad2030a13ecd61ac6ef7d6973981c1f62c7</header-checksum> <location href="repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck"/> <timestamp>1587594156</timestamp> <size>477385</size> <open-size>259956</open-size> <header-size>1200</header-size> </data> ---- 8< ---- 2. make dnf synchronize repodata cache (dnf makecache) Actual results: plain comps.xml is downloaded: $ ls -la /var/cache/dnf/fedora-558931b5e76b51a7/repodata/*comps* -rw-r--r--. 1 root root 1772199 5. lis 09.16 /var/cache/dnf/fedora-558931b5e76b51a7/repodata/961f955de96f1489e49012cfba125d771d2b4dce6cb31678aea644dffcfb3935-comps-Everything.x86_64.xml Expected results: compressed file (*.zck preferably) is downloaded Additional info: rpmfusion repositories workaround this using comps.xml.xz in group_gz metadata
*** Bug 1904364 has been marked as a duplicate of this bug. ***
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle. Changing version to 34.
This message is a reminder that Fedora Linux 34 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '34'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 34 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 34 entered end-of-life (EOL) status on 2022-06-07. Fedora Linux 34 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. Thank you for reporting this bug and we are sorry it could not be fixed.
I am sorry, but the issue is not yet resolved. I suggest that we should change the distribution rather to add the support of the new type of metadata.
May be little bit information around. Comps group were asipped as `group` type (uncompressed xml), and group_gz (compressed xml by gz). Other metadata in repositories were shipped only as compressed (e.g. `primary`). Then distribution added zchung for various metadata type (`primry_zch`). Then the compresson of metadata was changed. `primary` type remained unchanged refering to file compressed by another algorithm but `groups` compressed variant is shipped as `group_xz` - new type of metadata unsupported by DNF. I am proposing to ship `group_xz` as `group_gz` to restore functionality.
The strange thing is that the Fedora updates repos do use group_gz but the Fedora release repos use group_xz Anyway, it appears that DNF may already support transparently compressed "group" by virtue of this line https://github.com/rpm-software-management/dnf/blame/a3ececb7ddb7b5ee8911035e50b39aa97302cfe2/dnf/base.py#L703 By the point it gets to decompression, DNF has no idea which file it got from repomd.xml anyway, because it's abstracted by getCompsFn(). Decompression-or-not is decided by the file extension rather than what key it came from. If this support is fairly reliable and goes back sufficiently far in history, my suggestion is to just use the "group" tag with compressed content and throw "group_gz" and "group_xz" back into the abyss. However it looks like legacy yum might choke on this, which means that generating EL6/EL7 repos with this strategy may be problematic. Perhaps the --compatibility flag could be extended to handle this case? If this absolutely cannot be changed for fear of breakage, I would at least suggest opting into the new strategy instead of doubling down on the inconsistency of the old way.
Well, it looks like that createrepo_c is responsible for creation of data that cannot be loaded by DNF. Reproducer: createrepo_c --xz -g 721f55ca56291adc1d1a566c06716b12c2373e57b213c6da778a9349e17e312a-comps-Everything.x86_64.xml . According to discussion on Fedora-releng, the behavior was changed by https://github.com/rpm-software-management/createrepo_c/commit/a3c8f027af1f15c9e93c34a5ba3b82e8a3e630f9. We need to discuss whether we should modify createrepo_c or DNF to fix the current state.
Personally my inclination is to fix this in createrepo_c because the root of the issue is that the strategy that createrepo_c is using isn't generalizable. If the repo metadata used a consistent approach instead of handling comps differently by using a suffix, this would actually work fine. Question: Does EL6 and EL7 "yum" support XZ compressed metadata to begin with? If not, we can safely assume that any repo using those compression types can simply compress comps.xml the same way and put it under the "group" id.
Looks like it does, unfortunately, so that idea isn't helpful.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.
The issue is resolved by createrepo_c-1.0 which generates metadata differently - only compresses comps by default.