Bug 1992209

Summary:	createrepo --update removes the module.yaml from the repodata if "modules.yaml" is not in the directory
Product:	Red Hat Enterprise Linux 8	Reporter:	jcastran
Component:	createrepo_c	Assignee:	amatej
Status:	CLOSED ERRATA	QA Contact:	Jan Blazek <jblazek>
Severity:	high	Docs Contact:	Mariya Pershina <mpershin>
Priority:	high
Version:	8.4	CC:	amatej, kwalker, page.cal, pkratoch
Target Milestone:	beta	Keywords:	Reopened, Triaged
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	createrepo_c-0.17.7-4.el8	Doc Type:	Bug Fix
Doc Text:	.Running `createrepo_c --update` on a modular repository now preserves modular metadata in it Previously, when running the `createrepo_c --update` command on an already existing modular repository without the original source of modular metadata present, the default policy was to remove all additional metadata including modular metadata from this repository, which, consequently, broke it. To preserve metadata, it required running the `createrepo_c --update` command with the additional `--keep-all-metadata` option. With this update, you can preserve modular metadata on a modular repository by running `createrepo_c --update` without any additional option. To remove additional metadata, you can use the new `--discard-additional-metadata` option.	Story Points:	---
Clone Of:
Clones:	2055032 (view as bug list)		Environment:
Last Closed:	2022-05-10 13:57:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2055032

Description jcastran 2021-08-10 18:03:37 UTC

Description of problem:
The latest update of createrepo now grabs a modules.yaml present in the directory and adds it automatically to the repodata, saving the previously required step of using modifyrepo.

If this unzipped modules.yaml file is removed, and createrepo --update is run, the modularity is removed from the repodata.

modules.yaml must remain in the repository or createrepo, with or without --update, will remove the *-modules.yaml.gz from the repodata. 

Version-Release number of selected component (if applicable):
createrepo_c-0.16.2-2.el8.x86_64

How reproducible:
Everytime

Steps to Reproduce:
~~~~~~~~~~~~~~~~ Build repodata with a modules.yaml file ~~~~~~~~~~~~~~~~
[root@r8 ~]# gunzip /var/cache/dnf/rhel-8-for-x86_64-appstream-rpms-9d3886b51bb367d7/repodata/e7eecb4464237e9f35c4c2cbf4c3b502bc0bff1fff6b557715a128782e1ddb05-modules.yaml.gz -c > modules.yaml
[root@r8 ~]# ls
bash-4.4.20-1.el8_4.x86_64.rpm  modules.yaml

[root@r8 ~]# createrepo .
Directory walk started
Directory walk done - 1 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished

[root@r8 ~]# ls repodata/
490c178630c4236c27cba1fae8d2c2adafcb24dc6a6419e63e7e94d4a26a48c7-filelists.xml.gz  b9b4df684ee0e2585ef547bd5959877cbd301435096aa9bf384a6f0a3f1886b2-primary.sqlite.bz2
4e74d1470408e76d24d92db559b95915a84ad73bcb299338f64b3a2964b19e6f-primary.xml.gz    cd02b2dda28b7438e3d49348903b4c3de47c312f3b4b3d77dd7a8c0be3b0aaf0-filelists.sqlite.bz2
80e02b156ca07d57b082ee3231fbae7f53653931943a4ce73ee8c9914b03b3e0-modules.yaml.gz   e3691ee3f913c946f52e680cc4cd76e745354ae2846755cc390a96a715c29845-other.xml.gz
a04cce7cacf3365758218c1f497d89b681f57d18e389672383e83583acc336a9-other.sqlite.bz2  repomd.xml

~~~~~~~~~~~~~~~~ remove the original modules.yaml file from the directory and update the repodata ~~~~~~~~~~~~~~~~
[root@r8 ~]# mv modules.yaml /tmp

[root@r8 ~]# createrepo --update .
Directory walk started
Directory walk done - 1 packages
type modules added to list from path: ./repodata/80e02b156ca07d57b082ee3231fbae7f53653931943a4ce73ee8c9914b03b3e0-modules.yaml.gz
Loaded information about 1 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished

~~~~~~~~~~~~~~~~ *modules.yaml.gz is gone ~~~~~~~~~~~~~~~~
[root@r8 ~]# ls repodata/
490c178630c4236c27cba1fae8d2c2adafcb24dc6a6419e63e7e94d4a26a48c7-filelists.xml.gz  b9b4df684ee0e2585ef547bd5959877cbd301435096aa9bf384a6f0a3f1886b2-primary.sqlite.bz2    repomd.xml
4e74d1470408e76d24d92db559b95915a84ad73bcb299338f64b3a2964b19e6f-primary.xml.gz    cd02b2dda28b7438e3d49348903b4c3de47c312f3b4b3d77dd7a8c0be3b0aaf0-filelists.sqlite.bz2
a04cce7cacf3365758218c1f497d89b681f57d18e389672383e83583acc336a9-other.sqlite.bz2  e3691ee3f913c946f52e680cc4cd76e745354ae2846755cc390a96a715c29845-other.xml.gz

[root@r8 ~]# grep module repodata/repomd.xml  -c
0

~~~~~~~~~~~~~~~~ Move the unzipped modules.yaml back in and it's included again ~~~~~~~~~~~~~~~~
[root@r8 ~]# mv /tmp/modules.yaml .

[root@r8 ~]# createrepo --update .
Directory walk started
Directory walk done - 1 packages
Loaded information about 1 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished

[root@r8 ~]# ls repodata/
490c178630c4236c27cba1fae8d2c2adafcb24dc6a6419e63e7e94d4a26a48c7-filelists.xml.gz  b9b4df684ee0e2585ef547bd5959877cbd301435096aa9bf384a6f0a3f1886b2-primary.sqlite.bz2
4e74d1470408e76d24d92db559b95915a84ad73bcb299338f64b3a2964b19e6f-primary.xml.gz    cd02b2dda28b7438e3d49348903b4c3de47c312f3b4b3d77dd7a8c0be3b0aaf0-filelists.sqlite.bz2
80e02b156ca07d57b082ee3231fbae7f53653931943a4ce73ee8c9914b03b3e0-modules.yaml.gz   e3691ee3f913c946f52e680cc4cd76e745354ae2846755cc390a96a715c29845-other.xml.gz
a04cce7cacf3365758218c1f497d89b681f57d18e389672383e83583acc336a9-other.sqlite.bz2  repomd.xml

Expected results:
createrepo --update should not remove *-modules.yaml.gz from repodata, EVEN IF, the unzipped modules.yaml is gone. 

Additional info:
The ability for createrepo to automatically include the modules.yaml came from 

    Bugzilla 1795936 – [RFE] createrepo_c should be able to handle modules information
    https://bugzilla.redhat.com/show_bug.cgi?id=1795936

Comment 1 amatej 2021-08-16 10:06:26 UTC

I think this behavior is expected. --update controls whether createrepo_c recalculates metadata from rpm files or if it reuses already present metadata (from primary, filelists..).

If you want to preserve the *-modules.yaml.gz during an update even when their source was removed from the repo you can use --keep-all-metadata option.

Perhaps we could describe this more explicitly in the man page?

Comment 2 jcastran 2021-08-16 13:00:43 UTC

RHEL 7 and below don't have that option. The --update would leave any additional repodata files in place.

Can we instead change "--update" to use the --keep-all-metadata option by default? This would match the behaviour customers have used for years and help to avoid more differences in the steps for maintaining repositories of RHEL 5-7 and RHEL 8.

Comment 3 amatej 2021-08-17 05:40:26 UTC

We have discussed it and we most likely don't want to do such a change at this point.

Maybe in the future if we ever do a bigger revision and clean up of the createrepo_c options.

Comment 4 amatej 2021-09-06 08:11:18 UTC

I am closing this as NOTABUG because we currently don't have any plans to change this.

If there is some new development or some new information please feel free to reopen this bug.

Comment 7 amatej 2021-09-23 08:16:23 UTC

I made an upstream PR that proposes the change of using --keep-all-metadata by default and adds --discard-additional-metadata option to cover the opposite use case.
https://github.com/rpm-software-management/createrepo_c/pull/287
Tests update:
https://github.com/rpm-software-management/ci-dnf-stack/pull/1037

> RHEL 7 and below don't have that option. The --update would leave any additional repodata files in place.
I have tried it and while it does leave the additional repodata in place (in the repodata directory) it removes them from repomd.xml so I think they are technically no longer in the repo. To me it seems closer to the current createrepo_c behavior (with --keep-all-metadata off by default).
On the other hand we might want to change this because of modules.yaml.

Comment 9 Cal Page 2022-01-02 13:30:47 UTC

I ran into this modules.yaml when I tried to build a custom .iso for my client. In short, the .iso would contain some rpm's from RHEL 8, and the additional rpm's we are providing as part of our product.

The customer has air-gapped their computers for security reasons and can't use the over-the-internet method of installation.

As a work-around, I provide a custom entry in /etc/yum.repos.d that lets them loop mount the RedHat 8.5 iso and update from that.

Still, if you advertise that createrepo builds a repo, it should also build any needed yaml files.

Or perhaps, is there a way I can get past this that I am unaware.

Thank You
Cal Page

Comment 10 Cal Page 2022-01-05 16:05:19 UTC

So, the client wants me to DO SOMETHING, a work-around, whatever.

Trouble is, the modules.yaml has what looks like an encrypted front end name. I suspect this is part of your licensing system.

So, if my customer is asking me to HACK your licensing system, I'd have to say no. I'm a White Hack hacker ONLY.

So, can you please suggest a way I can move past this in accordance with your licensing system.

Best Regards,
Cal Page

Comment 11 amatej 2022-01-06 06:49:59 UTC

(In reply to Cal Page from comment #10)
> Trouble is, the modules.yaml has what looks like an encrypted front end
> name. I suspect this is part of your licensing system.

I am not sure what you mean here but I don't think there should be any encryption or licensing problems.
If you are referring to a name like: 3cc6e3f2bb6f513d28f0d50dd151a3ae4641af22e5963d193b4e73dc572378c0-modules.yaml.gz that is just a checksum + modules.yaml + compression suffix.

> So, can you please suggest a way I can move past this

If you are using modular rpms from some RHEL 8 repo you should take the modules.yaml from that repo as well.
In general createrepo_c cannot generate the modules.yaml file because it doesn't know how the user wants to arrange to modules (just like createrepo_c cannot generate groups.xml because it doesn't know what should each group look like and contain).

If you want to create your own modules we have some tools here: https://github.com/rpm-software-management/modulemd-tools




However unless I am misunderstanding all of this is not related to this bug (which is about changing default of --keep-all-metadata parameter).

Comment 19 errata-xmlrpc 2022-05-10 13:57:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (createrepo_c bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1866