Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2143497 - Can't perform incremental content exports in syncable format
Summary: Can't perform incremental content exports in syncable format
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.12.0
Hardware: All
OS: All
unspecified
urgent with 4 votes
Target Milestone: 6.13.0
Assignee: Partha Aji
QA Contact: Shweta Singh
URL:
Whiteboard:
: 2159491 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-11-17 03:30 UTC by Glenn Snead
Modified: 2023-09-19 04:30 UTC (History)
19 users (show)

Fixed In Version: pulpcore-3.21.6, pulpcore-3.22.2,tfm-rubygem-katello-4.7.0.11-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2175003 (view as bug list)
Environment:
Last Closed: 2023-05-03 13:22:42 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 35948 0 Normal Closed Need incremental export for syncable format 2023-02-14 13:48:22 UTC
Github pulp pulpcore issues 3313 0 None closed `UnexportableArtifactException` does not expect any arguments during the initialization 2023-03-29 19:35:01 UTC
Github pulp pulpcore issues 3413 0 None closed Provide start_version and end version for fs exporter. 2023-02-14 13:48:22 UTC
Red Hat Issue Tracker SAT-14971 0 None None None 2023-01-18 19:13:50 UTC
Red Hat Product Errata RHSA-2023:2097 0 None None None 2023-05-03 13:22:56 UTC

Description Glenn Snead 2022-11-17 03:30:39 UTC 
Description of problem:

Red Hat often has to build disconnected Red Hat content servers (CDNs) at disconnected clients to meet a vital disconnected issue where there are multiple organization's who can't or won't share a Satellite server and won't or can't use an upstream Satellite server.  What's more, the disconnected CDN can hold end-of-support software content that customers are still using and do not plan on migrating to a supported version for various reasons. 

Satellite 6.12 does allow full content exports in syncable format, the first Pulp 3 release to do so, but....

The hammer content-export --format option is only available for full content exports. Keep in mind that RHEL 8 AppStream and BaseOS are over 100 GB in size, and RHEL 9 won't be far behind. Then there are all of the other large repositories. Red Hat has nearly 6 TB of content at one client and that supports dozens of downstream Satellites, some of which can't be connected to Red Hat's upstream Satellite for various reasons.

I would like to have the "--format" option added to the hammer content-export incremental options list.

Version-Release number of selected component (if applicable): 6.12


How reproducible:

Steps to Reproduce:

1. hammer content-export incremental <library, repository, version> -h

The --format option is not present.


Actual results:

Usage:
    hammer content-export incremental version [OPTIONS]

Options:
 --async                                        Do not wait for the task
 --chunk-size-gb NUMBER                         Split the exported content into archives no greater than the specified size in
                                                gigabytes.
 --content-view[-id] VALUE/NUMBER               Name/Id of associated content view
 --destination-server VALUE                     Destination Server name
 --fail-on-missing-content                      Fails if any of the repositories belonging to this version are unexportable.
 --from-history-id NUMBER                       Export history id used for incremental export. If not provided the most recent
                                                export history will be used.
 --id NUMBER                                    Content view version id
 --lifecycle-environment[-id] VALUE/NUMBER      Name/Id of associated lifecycle environment
 --organization[-id|-title|-label] VALUE/NUMBER Name/Title/Label/Id of associated organization
 --version VALUE                                Filter versions by version number.
 -h, --help                                     Print help




Expected results:

Usage:
    hammer content-export incremental version [OPTIONS]

Options:
 --async                                        Do not wait for the task
 --chunk-size-gb NUMBER                         Split the exported content into archives no greater than the specified size in
                                                gigabytes.
 --content-view[-id] VALUE/NUMBER               Name/Id of associated content view
 --destination-server VALUE                     Destination Server name
 --fail-on-missing-content                      Fails if any of the repositories belonging to this version are unexportable.
 --format ENUM                                  Export formats. Choose syncable if content is to be imported via repository
                                                sync. Choose importable if content is to be imported via hammer content-import.

 --from-history-id NUMBER                       Export history id used for incremental export. If not provided the most recent
                                                export history will be used.
 --id NUMBER                                    Content view version id
 --lifecycle-environment[-id] VALUE/NUMBER      Name/Id of associated lifecycle environment
 --organization[-id|-title|-label] VALUE/NUMBER Name/Title/Label/Id of associated organization
 --version VALUE                                Filter versions by version number.
 -h, --help                                     Print help


Additional info:

Also, when a library export is performed, a sort of Content View is created...but it doesn't appear under the Content View list.
Comment 3 Partha Aji 2023-01-13 23:24:51 UTC 
Connecting redmine issue https://projects.theforeman.org/issues/35948 from this bug
Comment 4 Bryan Kearney 2023-01-14 00:03:14 UTC 
Upstream bug assigned to paji
Comment 5 Bryan Kearney 2023-01-14 00:03:16 UTC 
Upstream bug assigned to paji
Comment 6 Brad Buckingham 2023-01-23 17:37:42 UTC 
*** Bug 2159491 has been marked as a duplicate of this bug. ***
Comment 8 Vladimír Sedmík 2023-02-23 22:02:18 UTC 
Verified in 6.13.0 snap 11.

Tested the Library, Repository and Version exports with yum (custom, RedHat, KS) and file content types.

Yum content exports (which is the main use case) look good - on incremental exports only the newly added packages were exported.
File content exports have an issue - incremental exports still contain the full repo content. Not to block the main use case, the file issue will be tracked under https://bugzilla.redhat.com/show_bug.cgi?id=2172564

Another minor corner-case issue will be tracked under https://bugzilla.redhat.com/show_bug.cgi?id=2172845
Comment 10 juwatts 2023-03-29 14:50:39 UTC 
Moving this to Failed ON_QA due to

Steps:
1. Import few RedHat, custom and KS repos and enable these repos.
2. Sync the repos and try "hammer content-export complete" with option "--format syncable" for library, repositories and file content types.
3. Add another repository and try "incremental content-export" for the added repository.

Observation:
If export is attempted after changing "on-demand" download policy to "immediate" without re-syncing the repos, it will throw error.
Comment 11 Robin Chan 2023-03-29 15:16:26 UTC 
Requesting needsinfo from upstream developer dkliban, ggainey because the 'FailedQA' flag is set.
Comment 12 Grant Gainey 2023-03-29 19:35:01 UTC 
Backported and released the missing fix as core/3.21.6
Comment 13 Shweta Singh 2023-04-10 04:23:10 UTC 
Verified.

Version Tested: Satellite 6.13.0 Snap 17.0

Verification Steps:
1. Enable few RedHat repos and Custom repos.
2. Change the download policy of the repos from on_demand to immediate.
3. Sync the repos.
4. Do the complete content-export for repositories and library with command --format syncable.
5. Enable and few other repos and change the download policy as done earlier and sync the repos.
5. Do the incremental content-export with command --format syncable.
6. Verify that the incremental export should be done properly.

Observation:
Complete and incremental export is successfully completed.
Comment 16 errata-xmlrpc 2023-05-03 13:22:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097
Comment 17 Red Hat Bugzilla 2023-09-19 04:30:10 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.