Bug 1670651 - createrepo_c --keep-all-metadata doesn't keep all metadata
Summary: createrepo_c --keep-all-metadata doesn't keep all metadata
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: createrepo_c
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: amatej
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-30 02:54 UTC by Rohan McGovern
Modified: 2019-06-06 02:34 UTC (History)
7 users (show)

Fixed In Version: createrepo_c-0.13.2-2.fc30 createrepo_c-0.13.2-2.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-23 17:48:35 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Rohan McGovern 2019-01-30 02:54:47 UTC
Description of problem:

createrepo_c --keep-all-metadata --update doesn't keep all metadata as the option name suggests.  

Version-Release number of selected component (if applicable):

0.11.1-2.fc29

How reproducible:

100%

Steps to Reproduce:

1. Prepare/obtain a repo with modulemd, as in example:

$ find myrepo/
myrepo/
myrepo/repodata
myrepo/repodata/89ebbf3f135f25a1f600602207318f8b3fc2d0eb0851af5669b47b956abcfffe-primary.xml.gz
myrepo/repodata/4f9bf8da7d86c71fa4f7cb01ffe631ccf3cae6fedb22815beebbc56d30ebd5a8-modules.yaml.gz
myrepo/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
myrepo/repodata/f00d758e99c401ac7d5841770907ef7b3e86e89c8e035a212e693855856d88bf-other.xml.gz
myrepo/repodata/590354d60d7b4be2c1c40f5eaed3ac9209092dc6c94c72f2b9f63aa92f50c745-updateinfo.xml.gz
myrepo/repodata/fdbe8c908a8eb2a680801959112643e35792d39de87730015dd2233716befcee-filelists.xml.gz
myrepo/repodata/repomd.xml
myrepo/Packages
myrepo/Packages/r
myrepo/Packages/r/release-e2e-test-1.0.1640-1.el7.x86_64.rpm


2. Try to add sqlite to the repo with command: createrepo_c -d --update --keep-all-metadata ./myrepo/

3. Observe new contents of repo


Actual results:

sqlite files are added, but modulemd file is lost, as in example:

find myrepo/
myrepo/
myrepo/repodata
myrepo/repodata/9f7e68e69d7e2453caa3865a97f8165c1f34d089010c8eb00e7fef6396a6aafa-comps.xml.gz
myrepo/repodata/61f7528520042e94b2bb02568be957ecc311049a8d91ae71cb361a7cc705b1ec-filelists.sqlite.bz2
myrepo/repodata/06100c7a259671cb36cd84e23954fed4b79be03f626dd703ec0eb6dd56e85cde-filelists.xml.gz
myrepo/repodata/9fcb8e5b757cd06b8f20373d074d4864a10785f7c9ad0f0cf159bc798464c376-primary.xml.gz
myrepo/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
myrepo/repodata/83a86dc28a0ea6758f4abb5e88f4523153dafa9ea6fd8287c3db7e238b2559d4-other.sqlite.bz2
myrepo/repodata/fee28491e9dbeee98e83202de3d79ccb828ddf2b7c893ebeb8ce6cfa035496e3-primary.sqlite.bz2
myrepo/repodata/7bbb9aeda401d03d6c66ef21a3aadc9443273364533d841091db13140eaf4cee-other.xml.gz
myrepo/repodata/590354d60d7b4be2c1c40f5eaed3ac9209092dc6c94c72f2b9f63aa92f50c745-updateinfo.xml.gz
myrepo/repodata/repomd.xml
myrepo/Packages
myrepo/Packages/r
myrepo/Packages/r/release-e2e-test-1.0.1640-1.el7.x86_64.rpm



Expected results:

sqlite files are added, modulemd files are retained.


Additional info:

Current behavior is consistent with the help text which is "Keep groupfile and updateinfo from source repo during update", however the option name suggests keeping *all* metadata and this would be genuinely more useful.

The current behavior impacts Pulp, which attempts to add sqlite to an existing repo like this:

        pipe = subprocess.Popen('createrepo_c -d --update --keep-all-metadata '
                                '--local-sqlite '
                                '-s %(checksum_type)s --skip-stat %(content_dir)s' %
                                {'checksum_type': checksum_type, 'content_dir': self.content_dir},
                                shell=True, stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE)

The Pulp developers apparently took the '--keep-all-metadata' option name at face value and didn't realize this would discard most kinds of metadata from the repo, such as modulemd and productid files.  As a result, enabling publish of sqlite files in Pulp will disable publish of modulemd and productid files.

Comment 1 amatej 2019-04-11 08:58:47 UTC
Created a PR for this: https://github.com/rpm-software-management/createrepo_c/pull/147

Comment 2 Fedora Update System 2019-05-07 11:21:14 UTC
createrepo_c-0.13.2-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-91adfda805

Comment 3 Fedora Update System 2019-05-07 11:21:20 UTC
createrepo_c-0.13.2-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-06775cc842

Comment 4 Fedora Update System 2019-05-07 17:10:03 UTC
createrepo_c-0.13.2-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-06775cc842

Comment 5 Fedora Update System 2019-05-08 03:57:13 UTC
createrepo_c-0.13.2-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-91adfda805

Comment 6 Fedora Update System 2019-05-20 08:02:44 UTC
createrepo_c-0.13.2-2.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-c1840c618b

Comment 7 Fedora Update System 2019-05-20 08:02:51 UTC
createrepo_c-0.13.2-2.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-7a58cdd433

Comment 8 Fedora Update System 2019-05-21 01:59:27 UTC
createrepo_c-0.13.2-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-7a58cdd433

Comment 9 Fedora Update System 2019-05-21 04:53:29 UTC
createrepo_c-0.13.2-2.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-c1840c618b

Comment 10 Fedora Update System 2019-05-23 17:48:35 UTC
createrepo_c-0.13.2-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 11 Fedora Update System 2019-06-06 02:34:38 UTC
createrepo_c-0.13.2-2.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.