Bug 1683769

Summary:	dnf missing zchunk support
Product:	[Fedora] Fedora	Reporter:	Adam Williamson <awilliam>
Component:	dnf	Assignee:	Jaroslav Rohel <jrohel>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	dmach, jdieter, jmracek, jrohel, mblaha, mhatina, packaging-team-maint, pkratoch, rpm-software-management, vmukhame, yaneti
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	dnf-4.2.1-1.fc30 dnf-4.2.1-1.fc29	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-03-19 19:38:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Adam Williamson 2019-02-27 18:37:39 UTC

As discovered and discussed in investigation of the failed Fedora-Rawhide-20190227.n.0 compose:

https://pagure.io/dusty/failed-composes/issue/1529

with current versions of dnf, libdnf, createrepo_c and librepo, we wind up generating repositories containing a zchunk-compressed comps file with repomd type 'group_gz_zck', which libdnf `Repo.getCompsFn()` returns when asked. This is a problem because dnf `Base.read_comps()` calls `getCompsFn()` then tries to decompress and read whatever file is returned itself, but it does not have the ability to recognize or decompress zchunk files, so it winds up trying to read the compressed file as plain XML, and blowing up. This breaks composes.

There are various ways we can work around this in the short term, but making dnf actually capable of recognizing and decompressing zchunk files (or moving this functionality into libdnf, I guess) seems like it should be done.

Comment 1 Jonathan Dieter 2019-02-28 22:19:08 UTC

While this is definitely should be done, I believe the correct short-term fix is to have librepo only automatically return zchunked files for primary, filelists and other, since they are decompressed by libsolv.  All other file types should be returned in the format requested, with the zchunk versions only being returned if specifically requested.  There's a PR at https://github.com/rpm-software-management/librepo/pull/141 that implements this behavior.

Comment 2 Jaroslav Rohel 2019-03-04 12:28:43 UTC

PR:
https://github.com/rpm-software-management/dnf/pull/1341
https://github.com/rpm-software-management/libdnf/pull/690

Comment 3 Fedora Update System 2019-03-13 13:41:51 UTC

dnf-plugins-core-4.0.6-1.fc29 libdnf-0.28.0-1.fc29 libcomps-0.1.11-1.fc29 createrepo_c-0.12.2-1.fc29 librepo-1.9.5-1.fc29 dnf-4.2.1-1.fc29 dnf-plugins-extras-4.0.4-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-a03be2cebe

Comment 4 Fedora Update System 2019-03-13 13:42:45 UTC

dnf-plugins-core-4.0.6-1.fc30 libdnf-0.28.0-1.fc30 libcomps-0.1.11-1.fc30 createrepo_c-0.12.2-1.fc30 librepo-1.9.5-1.fc30 dnf-4.2.1-1.fc30 dnf-plugins-extras-4.0.4-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 5 Fedora Update System 2019-03-13 15:37:04 UTC

createrepo_c-0.12.2-1.fc30, dnf-4.2.1-1.fc30, dnf-plugins-core-4.0.6-1.fc30, dnf-plugins-extras-4.0.4-1.fc30, libcomps-0.1.11-1.fc30, libdnf-0.28.0-1.fc30, librepo-1.9.5-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 6 Jonathan Dieter 2019-03-13 23:06:40 UTC

I've tested libdnf-0.28.0-1.fc30 with a test repository with zchunked updateinfo.xml.zck, and dnf group list worked perfectly.  I also tested GNOME software, and it showed the update descriptions from the zchunked updatedinfo.xml.zck.  It looks like this is fixed now.

Comment 7 Fedora Update System 2019-03-14 18:51:34 UTC

createrepo_c-0.12.2-1.fc29, dnf-4.2.1-1.fc29, dnf-plugins-core-4.0.6-1.fc29, dnf-plugins-extras-4.0.4-1.fc29, libcomps-0.1.11-1.fc29, libdnf-0.28.0-1.fc29, librepo-1.9.5-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a03be2cebe

Comment 8 Fedora Update System 2019-03-15 13:09:37 UTC

createrepo_c-0.12.2-1.fc30 dnf-4.2.1-1.fc30 dnf-plugins-core-4.0.6-1.fc30 dnf-plugins-extras-4.0.4-1.fc30 libcomps-0.1.11-1.fc30 libdnf-0.28.0-1.fc30 librepo-1.9.5-1.fc30 swid-tools-0.7.3-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 9 Fedora Update System 2019-03-16 20:17:33 UTC

createrepo_c-0.12.2-1.fc30, dnf-4.2.1-1.fc30, dnf-plugins-core-4.0.6-1.fc30, dnf-plugins-extras-4.0.4-1.fc30, libcomps-0.1.11-1.fc30, libdnf-0.28.0-1.fc30, librepo-1.9.5-1.fc30, swid-tools-0.7.3-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 10 Fedora Update System 2019-03-19 19:38:43 UTC

createrepo_c-0.12.2-1.fc30, dnf-4.2.1-1.fc30, dnf-plugins-core-4.0.6-1.fc30, dnf-plugins-extras-4.0.4-1.fc30, libcomps-0.1.11-1.fc30, libdnf-0.28.0-1.fc30, librepo-1.9.5-1.fc30, swid-tools-0.7.3-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 11 Fedora Update System 2019-03-20 22:13:04 UTC

createrepo_c-0.12.2-1.fc29, dnf-4.2.1-1.fc29, dnf-plugins-core-4.0.6-1.fc29, dnf-plugins-extras-4.0.4-1.fc29, libcomps-0.1.11-1.fc29, libdnf-0.28.0-1.fc29, librepo-1.9.5-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2019-03-21 14:40:41 UTC

createrepo_c-0.12.2-1.fc29, dnf-4.2.1-1.fc29, dnf-plugins-core-4.0.6-1.fc29, dnf-plugins-extras-4.0.4-1.fc29, libcomps-0.1.11-1.fc29, libdnf-0.28.0-1.fc29, librepo-1.9.5-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.