Bug 1632961

Hi Perumal,

The behavior observed for content views is expected, per https://bugzilla.redhat.com/show_bug.cgi?id=1629060#c1 .

Chris,

Do you know if this is expected behavior for exporting of repositories?  (i.e. should the --since option work?)

Perumal,

Are you exporting a repo from Library, or from a CV?

Hello Chris,

We should be able to do it for Repo and also the content under Library Default organization view.

Regards 

Perumal

Perumal,

When I did the export at first, I observed the same behavior. However, it turned out that I forgot to delete my /var/lib/pulp/katello-export/<uuid>-incremental directory first, so the old RPMs were still there.

Can you try this?

* perform export using a date far in the past, like 2010-01-01T12:00:00Z. This should get all RPMs.

* delete export directory

* perform export again, but using date in the future, like 2019-01-01T12:00:00Z. This should create an empty repo, which shows that the date is being passed in.

Note that the --since date is based on when the package was added to the repo in Satellite via sync. For example, if a package was added to CDN on a Monday, and the customer synced on a Wednesday, they would need to put Tuesday as the --since date to pick up the new package. This would catch everything from Tuesday onward.

If you're still not able to get it to work just let me know. The --since flag is a bit finicky, we are working to improve the process in 6.5.

I have a customer who's run into this issue on a Satellite 6.8 server, and I was able to reproduce it on my Satellite 6.7 server (moya67, 10.10.182.42, /var/lib/pulp/katello-export).  So far, I haven't been able to find a way to prevent or work around the issue.

Red Hat Solutions Architect Rick Ring found that the --since option does appear to give incremental updates for the organization's "Default Organization View".  Here's an example of that command in use:

# hammer content-view version export-legacy --organization "MYORG" --content-view-id 1 --export-to-iso 1 --since 2021-04-04T18:41:11EDT

However, this approach does not make use of content-views or lifecycle environments (other than the default ones), so customers with strict version control policies would have to be more careful about when they sync their repositories.

(This workaround seems to work on both Satellite 6.7 and 6.8.)

This should be addressed with Satellite 6.10's re-work of the export routine with Pulp 3.

Info:  The new export/import does not use flag "--since" anymore.  Instead, it uses "incremental" export now.  Please find the below explanation:

"Exporting complete versions can be a very expensive operation on storage space and resources. Content View versions that have multiple Red Hat Enterprise Linux trees can occupy several gigabytes of the space on Foreman server.

You can use the Incremental Export functionality to help reduce demands on your infrastructure. Incremental Export exports only content that changes from the previously exported version. Generally, incremental changes are smaller than full exports. ln the following example, since version 1.0 has already been exported and the command targets version 2.0 for export."


Steps to test:

1. Spin up 2 Satellites (6.10 each)

#Export Satellite Portion

2. On satellite to export, import manifest
3. Administer -> Setting -> Search for "Default Red Hat Repository download policy" -> Set policy to "immediate"
4. Enable a several RH repos
5. Sync the repos
6. Create a cv and split up adding one repo from step 4
7. Publish cv
8. Run complete export command: hammer content-export complete version --id <cvv-id> --organization-id <org-id>
9. Verify that the export files are created in "/var/lib/pulp/exports/<organization name>/<cv name><export time stamp>

#Import Disconnected Satellite Portion

10. On the satellite to import the cv, import manifest that would have all the repos created in step 4
11. Set "Default Red Hat Repository download policy" to "immediate" just like step 3.
12. Set "Disconnected Mode" to "Yes" by going to Administer -> Setting -> Content Tab -> Disconnected Mode
13. SCP the files from the exported directory in step 8 to /var/lib/pulp/imports in the satellite in which you want to import
14. Provide write permission to the dir imported: chown -R pulp:pulp /var/lib/pulp/imports/<dir with import>
15. Verify that the files are present in /var/lib/pulp/imports/<dir with import>
16. Run the import command: hammer content-import version --organization-id <org-id> --path /var/lib/pulp/imports/<dir with import>
17. Verify that on the UI that the CV and it's content is all imported over.

# Back to Export Satellite portion to test incremental export

18. Add a second repo to the cv created in step 6 and publish to version 2.0
19. Run incremental export command: hammer content-export incremental version --id <cv-version-id-for-version-2.0> --organization-id <org-id>
20. Check the repos within the metadata.json file by running: cat <dir of exported content>|json_reformat
21. Compare the sizes of the tar.gz files between the complete and incremental version: ls -lh <dir of exported content>

# Back to Import Disconnected Satellite Portion

22. SCP the files from the incremental exported directory in step 19 to /var/lib/pulp/imports in the satellite in which you want to import
23. Provide write permission to the dir imported: chown -R pulp:pulp /var/lib/pulp/imports/<dir with incremental import>
24. Verify that the files are present in /var/lib/pulp/imports/<dir with import>
25. Run the import command: hammer content-import version --organization-id <org-id> --path /var/lib/pulp/imports/<dir with import>
26. Verify that on the UI that the CV and it's content is all imported over. 

Expected:
20) Should include all repos from step 18 as well as ["incremental": true] in the json
21) Exported tar.gz should have a smaller size than the complete export one
22) All content should be present in UI

Actual:
20) Does include all repos from step 18 as well as ["incremental": true] in the json
    },
    "incremental": true,
    "from_content_view_version": {
        "major": 1,
        "minor": 0
    },

21) Exported tar.gz has a smaller size than the complete export one
[root@]# ls -lh /var/lib/pulp/exports/Default_Organization/one_cv/2.0/2021-06-22T15-45-52-04-00/            <----complete export
total 1.2G
-rw-r--r--. 1 pulp pulp 1.2G Jun 22 15:47 export-78415e71-88d3-4732-bd40-c21dd42de1e0-20210622_1945.tar.gz
-rw-r--r--. 1 pulp pulp  333 Jun 22 15:47 export-78415e71-88d3-4732-bd40-c21dd42de1e0-20210622_1945-toc.json
-rw-r--r--. 1 root root 3.3K Jun 22 15:47 metadata.json
[root@]# ls -lh /var/lib/pulp/exports/Default_Organization/one_cv/2.0/2021-06-22T15-38-46-04-00/            <----Incremental export
total 26M
-rw-r--r--. 1 pulp pulp  26M Jun 22 15:38 export-39f44873-4bba-4138-92b9-bb94af483aa5-20210622_1938.tar.gz
-rw-r--r--. 1 pulp pulp  333 Jun 22 15:38 export-39f44873-4bba-4138-92b9-bb94af483aa5-20210622_1938-toc.json
-rw-r--r--. 1 root root 3.4K Jun 22 15:39 metadata.json


22) All content is be present in UI


Verified on 6.10_snap 5

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702