RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2056318 - createrepo_c creates two group definition compressed comps.xml.gz and uncompressed comps.xml in repodata
Summary: createrepo_c creates two group definition compressed comps.xml.gz and uncompr...
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: createrepo_c
Version: 9.1
Hardware: All
OS: Linux
unspecified
low
Target Milestone: rc
: 9.1
Assignee: David Cantrell
QA Contact: swm-qe
Mariya Pershina
URL:
Whiteboard:
Depends On: 1904360
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-21 00:38 UTC by Ameya Patil
Modified: 2023-08-31 08:47 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
.Running `createrepo_c` on local repositories generates duplicate `repodata` files When you run the `createrepo_c` command on local repositories, it generates duplicate copies of `repodata` files, one of the copies is compressed and one is not. There is no workaround available, however, you can safely ignore the duplicate files. The `createrepo_c` command generates duplicate copies because of requirements and differences in other tools relying on repositories created by using `createrepo_c`.
Clone Of:
Environment:
Last Closed: 2023-08-09 07:36:48 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-112860 0 None None None 2022-02-21 00:44:21 UTC

Internal Links: 2123593

Description Ameya Patil 2022-02-21 00:38:54 UTC
Description of problem:

Every time we run createrepo with -g option passing the group definition , It actually create two group files in repodata directory for group i.e. one compressed comps.xml.gz and other comps.xml.

The repomd.xml contain references to both these files
This however does not create any issue to DNF functionality , as I tested it with BaseOS repo, where even if I remove any one of the file by using modifyrepo command the dnf still fetches group definition regardless as long as any one file exist.

So this is this is just that createrepo is creating an addition file here for which the bug is raised. It does not necessarily create any functionality problem as per my testing.


Version-Release number of selected component (if applicable):
createrepo_c-libs-0.17.2-3.el8.x86_64
createrepo_c-0.17.2-3.el8.x86_64


How reproducible:
Every time we pass -g comps.xml during running createrepo

Steps to Reproduce:
1. Create a reposync of such a repository with which you can test the group functionality. Since BaseOS has group definition, I used it here.
 
 # dnf reposync --gpgcheck --download-metadata --repoid=rhel-8-for-x86_64-baseos-rpms --download-path /root/reposync/
 
2. Run createrepo with group by using -g option to pass the comps files to it.

 # createrepo -g /root/reposync/rhel-8-for-x86_64-baseos-rpms/comps.xml /root/reposync/rhel-8-for-x86_64-baseos-rpms/

3. We see here two file are generated i.e. dac5935a873f81b405354b478fafe4042a31aa9702e29a7a6ad5cd0c65e140e5-comps.xml and 9f4d7f28944e4206f5cc541486370745e11b1d886de3f6a2c7f3c0cc81418c4b-comps.xml.gz

~~~
# ls -l rhel-8-for-x86_64-baseos-rpms/repodata/
total 105608
-rw-r--r--. 1 root root 18330560 Jan 21 04:40 19d08569effc355478c7ba35e30ba89b5d017827f3f7230d677a49540a767ace-filelists.xml.gz
-rw-r--r--. 1 root root 38030852 Jan 21 04:41 2a9b9a7319595564ca556b9132b38f35154abc95bf596e8cc7a3f90d2699932b-primary.sqlite.bz2
-rw-r--r--. 1 root root 19651276 Jan 21 04:41 58c640def52a1d85224a7b68c28535786da9dece5a251ee11c04d159e3884b1b-filelists.sqlite.bz2
-rw-r--r--. 1 root root  4323504 Jan 21 04:40 6bcacda1b84b9385dc83d85efa2f8399e5261e98af9d5570a555c13e66354762-other.sqlite.bz2
-rw-r--r--. 1 root root  3978391 Jan 21 04:40 762bdceb3fc94c4339ab684fec88fffc1ac22b57b6b82257ee47e47fb6b2fb90-other.xml.gz
-rw-r--r--. 1 root root 23420984 Jan 21 04:40 994dad4b80885767923411c2cadbcce622cd7b66bb4aa6b03fa25fa47f8b92a7-primary.xml.gz
-rw-r--r--. 1 root root    76624 Jan 21 04:40 9f4d7f28944e4206f5cc541486370745e11b1d886de3f6a2c7f3c0cc81418c4b-comps.xml.gz
-rw-r--r--. 1 root root   309207 Jan 21 04:38 dac5935a873f81b405354b478fafe4042a31aa9702e29a7a6ad5cd0c65e140e5-comps.xml
-rw-r--r--. 1 root root     3885 Jan 21 04:41 repomd.xml
~~~

Actual results:
createrepo is creating two seperate files for group defination in the repodata and both the files are updated in repomd.xml file.

Expected results:
createrepo should be creating one of the group definition files.

Additional info:
Both the files entries are made into `repomd.xml` file.
However, "dnf grouplist -v" works even if I remove the repodata with any one of the files i.e. xml one or the compressed xml one, so functionality wise this will not cause any issue.

I tested by using modifyrepo to remove the compressed file and keeping the uncompressed version dnf was able to fetch the group metadata as well as doing the opposite i.e. using modifyrepo to remove the uncompressed file and dnf was still able to fetch group metadata.

 # modifyrepo --verbose --remove group_gz /root/reposync/rhel-8-for-x86_64-baseos-rpms/repodata

Or 

 # modifyrepo --verbose --remove group /root/reposync/rhel-8-for-x86_64-baseos-rpms/repodata

Comment 1 amatej 2022-02-21 13:25:35 UTC
Since this is a change in behavior would it be ok to target 9.1 for the fix?

Comment 3 Jaroslav Mracek 2022-07-12 10:25:46 UTC
Well I would be very careful with the change. It is true that DNF downloads only one of them (preferentially the compress one) and uncompressed is created later on, but it doesn't mean that it is a non braking change. We can provide a new option that will ensure the compatibility and keep the current default behavior. Or we can change the default behavior and deploy the change to Fedora first and then to RHEL to ensure that we discovered all side affects. But it means to postpon the release of the change to 9.3 We have to consider legacy user cases where the new createrepo will create content for old RHELs therefore we have to investigate acompatibility of the change with yum 3 (RHEL 7) first.

Comment 4 Kyle Walker 2022-07-12 21:11:55 UTC
I will add another voice in support of not making this change mid-RHEL. This would be very prone to unnecessary breakage in various tooling that relies on one or the other. If this is something that we would pursue at all, I would say that it _must_ go into Fedora first, and then have a lengthy validation period. Likely well beyond the RHEL 9 timeframe.

Comment 5 Daniel Alley 2022-09-02 02:56:28 UTC
Satellite currently expects the uncompressed variant to be there, which is an easy fix, but worth considering.  I concur with Jaroslav and Kyle that it shouldn't happen in 9.1.  9.3 might be reasonable, I can follow up on that in the future.

Comment 6 Daniel Alley 2023-01-06 21:26:07 UTC
Will the metadata tagged as "group" in repomd.xml disappear completely, with consumers expected to look for "group_gz" or "group_xz", or will "group_gz" and "group_xz" disappear, with "group" being compressed under any supported compression type?

Comment 7 Daniel Alley 2023-01-09 13:52:25 UTC
Keep in mind, nothing will actually change for RHEL repos since RHEL doesn't use createrepo_c to generate their repos.  Only the uncompressed groups metadata variant exists currently and that won't change.

Comment 8 Jaroslav Mracek 2023-01-09 16:32:35 UTC
It looks like that the request could be blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1904360. If we will remove `group` type and distribution will exchange `group_gz` by `group_xz` then we have a disaster.

Comment 9 Daniel Alley 2023-01-09 17:16:29 UTC
As per my comment here (https://bugzilla.redhat.com/show_bug.cgi?id=1904360#c7) I suggest that Fedora and RHEL should use compressed metadata with the "group" metadata name, which ought to work fine (I haven't tried it, but the logic of the code should handle it).

Whether createrepo_c can adopt that change is questionable. I believe as mentioned legacy yum (ie. EL6, EL7) may not work with such repos. However, we could either restore the old behavior under the --compatibility flag, or if that is too aggressive we could make it more opt-in.

Comment 10 Daniel Alley 2023-01-09 17:24:08 UTC
>However, we could either restore the old behavior under the --compatibility flag, or if that is too aggressive we could make it more opt-in.

I meant make the new behavior opt-in, as we can't assume that anyone who wants to generate EL6/EL7 repos is already using the --compatibility flag, even though it would be really convenient to just shove all legacy compatibility behaviors under that flag and in a vacuum that would absolutely be my preference.

Comment 12 Jaroslav Mracek 2023-03-20 12:08:34 UTC
The issue was created by https://github.com/rpm-software-management/createrepo_c/commit/a3c8f027af1f15c9e93c34a5ba3b82e8a3e630f9 version 0.13.0, therefore it created a regression in RHEL8.

I recommend to develop two solution.

For upstream and RHEL10:
Create only `group` type metadata (compressed (various compressions), or uncompressed). It means no `group_gz` or `group_xz`

This solution will unify behavior with standard metadata types (primary, filelists, ...).
The solution is not compatible with RHEL7 and YUM version 3 because it is unable to open compressed `group.xml`. Therefore uncompressed `group` file is required to create RHEL7 compatible repository (it might require an additional option).

AC: By default createrepo_c must not create `group_gz` or `group_xz` and even with `--xz` option used to modified compression algorith.
The new metadata must work correctly with DNF version from RHEL 8.0 GA.
Documentation must describe how to create RHEL7 compatible metadata


For RHEL8/9 there are 3 alternative solution:

A) (not compatible with RHEL7)
We should deliver a solution to not generate `group_xz` metadata when the `--xz` option is used. CentOS Stream distribution is affected with this issue.

I suggest to generate `group` type with compressed `group.xml` when `--xz` is used. No change in default behavior (`group_gz` and `group` will be kept).

AC: By default createrepo_c must create `group_gz` and `group`
 - With `--xz` option generate compressed `group` type of metadata.
 - The new metadata must work correctly with DNF version from RHEL 8.0 GA.
 - Documentation must mention that metadata generated with `--xz` are not compatible with RHEL7 (I believe that they were not compatible with RHEL7 anyway before the change for other metadata types)

B) (compatible with RHEL7)
We should deliver a solution to not generate `group_xz` metadata when the `--xz` option is used. CentOS Stream distribution is affected with this issue.

I suggest to generate `group_gz` type with compressed `group.xml` (by xz compression) and uncompressed `group` type when `--xz` is used. No change in default behavior (`group_gz` and `group` will be kept).

AC: By default createrepo_c must create `group_gz` and `group`
 - With `--xz` option generate compressed `group_gz` and uncompressed `group` type of metadata .
 - The new metadata must work correctly with DNF and YUM from RHEL7 and RHEL8.


C) (only documentation update)
Document that option `--xz` will generate compressed group metadata (`group_xz` type) unusable by anyone.

Comment 13 Daniel Alley 2023-03-20 18:22:55 UTC
>I suggest to generate `group_gz` type with compressed `group.xml` (by xz compression) and uncompressed `group` type when `--xz` is used. No change in default behavior (`group_gz` and `group` will be kept).

I'm not sure I understand what you're saying here, is it that "group_gz" would be compressed with xz compression?  If so I'm not sure I like that - "group_gz" should be "gz"-compresessed if it's kept around at all. What might make sense (and maybe this is what you meant?) is, produce all xz-compressed metadata including the "group" metadata, but additionally produce "group_gz" which is gzip-compressed.  Yum will prefer the latter so if it is present the former will be ignored (so it shouldn't break). DNF should be changed to not look for "group_gz" and instead pick up the "group" metadata which is compressed the same way everything else is.

Does that make sense?

Comment 14 Jaroslav Mracek 2023-03-24 07:05:14 UTC
Update for Comment 12:

RHEL8/9 (decision) - We will document the issue as KNOWN ISSUE without updating createrepo_c package.

Comment 15 Jaroslav Mracek 2023-08-09 07:36:48 UTC
The bug cannot be fixed during lifecycle of RHEL therefore I am closing it. We will document the issue as a known issue.


Note You need to log in before you can comment on or make changes to this bug.