Bug 2226950

Summary:	Size of a "Complete Library export" can be double the consumption of /var/lib/pulp/media/artifact/
Product:	Red Hat Satellite	Reporter:	Sayan Das <saydas>
Component:	Pulp	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED ERRATA	QA Contact:	sganar
Severity:	high	Docs Contact:
Priority:	high
Version:	6.13.3	CC:	ahumbe, dalley, dkliban, ggainey, hyu, iballou, jalviso, jpasqual, keiwilli, mjia, osousa, paji, pcreech, rchan, rlavi, vijsingh
Target Milestone:	6.14.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	pulpcore-3.28.8, pulpcore-3.23.13, pulpcore-3.22.12, pulpcore-3.21.15, pulpcore-3.18.25, pulpcore-3.16.19	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2227913 2231504 2257297 (view as bug list)		Environment:
Last Closed:	2023-11-08 14:20:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sayan Das 2023-07-27 06:14:05 UTC

Description of problem:

Size of a "Complete Library export" can be double the consumption of /var/lib/pulp/media/artifact/  when the same base repositories are part of multiple CVs and CCVs ( i.e. multiple copies of the same repo and artifacts exist).

See https://github.com/pulp/pulpcore/issues/4159 for more info.

Version-Release number of selected component (if applicable):

Satellite 6.13 


How reproducible:

Undetermined at present but possible to reproduce in the customer's environment. 


Steps to Reproduce: 

Probably:

1. Enable and sync these two repos with immediate download policies

- Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8.8
- Red Hat Enterprise Linux 8 for x86_64 - BaseOS Kickstart 8.8

2. Add them to a CV and publish a new version. 

3. Make 20 - 50 copies of that CV and Publish a new version for all of them.

4. Check the disk space consumption of  /var/lib/pulp/media/artifact/ directory.

5. Perform a full library export for that organization.

6. Check the size of export tar in the exported directory. 


Actual results:

Size noted in Step 6 is much bigger ( almost double ) than the size noted in Step 4.


Expected results:

The exported archive should never exceed the total size of artifacts downloaded in pulp


Additional info:

It might not be 100% possible to reproduce the issue with just RPM repos and the reproducer may need to include a file type repo or a docker type repo.

Comment 7 Robin Chan 2023-07-29 16:04:45 UTC

The Pulp upstream bug status is at closed. Updating the external tracker on this bug.

Comment 14 sganar 2023-09-25 13:45:12 UTC

Verified.

Tested on Satellite 6.14.0 Snap 17.0
python39-pulpcore-3.22.15-1.el8pc.noarch

Verification Steps:
1. Enable and sync few Red Hat repositories(Set the download policy to "Immediate") also create a file type or docker type repositories and sync them.
2. Add them to CV, publish and promote it.
3. Make 20-50 copies of the CV and Publish a new version for all of them.
4. Check the disk space consumption of var/lib/pulp/media/artifact/ directory.
5. Perform a full library export for that organization.
6. Check the size of exported tar file present in /var/lib/pulp/exports/Default_Organization/Export-Library/* directory.
7. Compare consumption in Step 5 and Step 7, size of tar file in Step 7 should not exceed the one in Step5

Observation:
The size of the exported tar file created in path /var/lib/pulp/exports/Default_Organization/Export-Library/* is not more than the disk space consumption of var/lib/pulp/media/artifact/ directory.

Comment 15 Daniel Alley 2023-10-12 00:02:57 UTC

*** Bug 2238914 has been marked as a duplicate of this bug. ***

Comment 16 Daniel Alley 2023-10-12 00:06:32 UTC

*** Bug 2238915 has been marked as a duplicate of this bug. ***

Comment 19 Ian Ballou 2023-10-30 20:53:35 UTC

A hotfix is now available for Satellite 6.11.5 on RHEL 7 and RHEL 8. Please contact Red Hat support for installation instructions.

Comment 22 errata-xmlrpc 2023-11-08 14:20:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818