Bug 2212523

Summary: Content-export incremental with syncable format based does not include productid file into repodata directory
Product: Red Hat Satellite Reporter: Joniel Pasqualetto <jpasqual>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: VERIFIED --- QA Contact: Vladimír Sedmík <vsedmik>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.12.4CC: ahumbe, dalley, dkliban, ggainey, jkwek, mkalyat, pcreech, rchan, rlavi, vsedmik, wpinheir
Target Milestone: 6.14.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulpcore-3.22.8-1, pulpcore-3.21.11-1, pulpcore-3.18.21-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joniel Pasqualetto 2023-06-05 19:37:50 UTC 
Description of problem:

Generating an incremental export with syncable format, generates an export with a missing file on the repodata directory.


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create a CV with a repository that contains a productid (example repo: Red Hat Ansible Engine 2 for RHEL 8 x86_64 RPMs )
2. Do a complete export of the CV:

 hammer content-export complete version --id 41 --format syncable

3. Create a new version of the CV
4. Export CV again, this time an incremental:

~~~
hammer content-export incremental version --id 42 --format syncable
~~~

Note the files on the repodata directory:

~~~
# ls |sort
2ad01aecc583cdede5a1489ee50efd97c48b923b586d24fb83b42f5c9103b85b-other.xml.gz
5ac9d773403beea431e1d08255b782b8bb38e61cd458254bb0cbde907f444a77-updateinfo.xml.gz
641203f1aa0962bbb1af40e20142c824e669323aaee862acd3eea80e046ccc92-primary.xml.gz
67120121d59a4351342ec001ccc4e7ad4520175029770bbda447c0186ed9f1e9-filelists.sqlite.bz2
8e91c89c9670575d4123bc37cd0cfb9e1f7a57a9b9b93c2491a67678fa7ec9d4-filelists.xml.gz
a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
ee7bbcac9b13fa0a8fe3aa083e4200c5391c945b76feb59e887c4d2e36944371-other.sqlite.bz2
f8d2107d953c43ede24bf85033c8ca025f1b32194e99b3919231ab0ce35c217a-primary.sqlite.bz2
repomd.xml
~~~

Looking inside repomd.xml, there will be a reference to file of data type "productid" which is missing:

~~~
# grep productid repomd.xml -A2
  <data type="productid">
    <checksum type="sha256">55a5ce52b4fe56961ccab8f58d2ac36230d09491d627e09f685d4b2ddd50e55b</checksum>
    <location href="repodata/7034ae03-a219-4911-9337-2118fa5768e8"/>
~~~

Actual results:
productid file is missing from repodata

Expected results:
A complete export of the repodata. An incremental export should only affect artifacts and not metadata.


Additional info:
Comment 5 Vladimír Sedmík 2023-07-31 14:06:03 UTC 
Verified in 6.14.0 snap 9 (python39-pulpcore-3.22.9-1.1.el8pc.noarch)

Now the incremental syncable exports contain the productid file:

1) Create and sync (immediate policy) the ansible-2-for-rhel-8-x86_64-rpms repo, add it to a CV, publish.

2) Run the complete export of the CV:
[root@sat ~]# hammer content-export complete version --id 5 --format syncable
[..................................................................................] [100%]
Generated /var/lib/pulp/exports/BZ-2212523/CV-1/1.0/2023-07-31T07-40-44-04-00/metadata.json

3) Publish new CV version and run incremental export of it:
[root@sat ~]# hammer content-export incremental version --id 6 --format syncable
[..................................................................................] [100%]
Generated /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/metadata.json

4) Check the productid in present the incremental export:
[root@sat ~]# ll /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/
total 5048
-rw-r--r--. 3 pulp pulp   21562 Jul 31 05:50 2ad01aecc583cdede5a1489ee50efd97c48b923b586d24fb83b42f5c9103b85b-other.xml.gz
-rw-r--r--. 3 pulp pulp    9190 Jul 31 05:50 5ac9d773403beea431e1d08255b782b8bb38e61cd458254bb0cbde907f444a77-updateinfo.xml.gz
-rw-r--r--. 3 pulp pulp    8065 Jul 31 05:50 641203f1aa0962bbb1af40e20142c824e669323aaee862acd3eea80e046ccc92-primary.xml.gz
-rw-r--r--. 3 pulp pulp 2411759 Jul 31 05:50 67120121d59a4351342ec001ccc4e7ad4520175029770bbda447c0186ed9f1e9-filelists.sqlite.bz2
-rw-r--r--. 3 pulp pulp    2139 Jul 31 05:50 7034ae03-a219-4911-9337-2118fa5768e8                                                    <======= here
-rw-r--r--. 3 pulp pulp 2553565 Jul 31 05:50 8e91c89c9670575d4123bc37cd0cfb9e1f7a57a9b9b93c2491a67678fa7ec9d4-filelists.xml.gz
-rw-r--r--. 3 pulp pulp     124 Jul 31 05:50 a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml
-rw-r--r--. 3 pulp pulp  109732 Jul 31 05:50 ee7bbcac9b13fa0a8fe3aa083e4200c5391c945b76feb59e887c4d2e36944371-other.sqlite.bz2
-rw-r--r--. 3 pulp pulp   27208 Jul 31 05:50 f8d2107d953c43ede24bf85033c8ca025f1b32194e99b3919231ab0ce35c217a-primary.sqlite.bz2
-rw-r--r--. 3 pulp pulp    4134 Jul 31 05:50 repomd.xml

5) Check it matches the repomd:
[root@sat ~]# cat /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/repomd.xml | grep productid -A 5
  <data type="productid">
    <checksum type="sha256">55a5ce52b4fe56961ccab8f58d2ac36230d09491d627e09f685d4b2ddd50e55b</checksum>
    <location href="repodata/7034ae03-a219-4911-9337-2118fa5768e8"/>
    <timestamp>1602714155</timestamp>
    <size>2139</size>
  </data>
[root@sat ~]# sha256sum /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/7034ae03-a219-4911-9337-2118fa5768e8
55a5ce52b4fe56961ccab8f58d2ac36230d09491d627e09f685d4b2ddd50e55b  /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/7034ae03-a219-4911-9337-2118fa5768e8

6) Check it's the same as the complete export:
[root@sat ~]# diff -s \
> /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/7034ae03-a219-4911-9337-2118fa5768e8 \
> /var/lib/pulp/exports/BZ-2212523/CV-1/1.0/2023-07-31T07-40-44-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/7034ae03-a219-4911-9337-2118fa5768e8
Files /var/lib/pulp/exports/BZ-2212523/CV-1/2.0/2023-07-31T07-43-16-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/7034ae03-a219-4911-9337-2118fa5768e8 and /var/lib/pulp/exports/BZ-2212523/CV-1/1.0/2023-07-31T07-40-44-04-00/content/dist/layered/rhel8/x86_64/ansible/2/os/repodata/7034ae03-a219-4911-9337-2118fa5768e8 are identical