Bug 2040870 - Error to import rhel7 kickstart on disconnected satellite
Summary: Error to import rhel7 kickstart on disconnected satellite
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.1
Hardware: Unspecified
OS: Unspecified
high vote
Target Milestone: 6.11.0
Assignee: satellite6-bugs
QA Contact: Lai
Depends On:
TreeView+ depends on / blocked
Reported: 2022-01-14 19:58 UTC by Joniel Pasqualetto
Modified: 2022-06-23 14:40 UTC (History)
9 users (show)

Fixed In Version: pulpcore-3.16.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2059393 (view as bug list)
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github pulp pulpcore issues 2192 0 None closed PulpImport/Export of kickstart repos with subrepos broken 2022-02-22 17:10:09 UTC

Description Joniel Pasqualetto 2022-01-14 19:58:06 UTC
Description of problem:
Exporting RHEL 7 kickstart repository from a connected Satellite and trying to import it on a disconnected one fails with error "Error: ContentArtifact matching query does not exist."

Version-Release number of selected component (if applicable):
Satellite 6.10.1

How reproducible:
Always, if the repository does not exists.

Steps to Reproduce:
1. Sync the kickstart repository using immediate policy
2. Create a CV containing only this repository
3. Export it, using hammer
4. Copy the export directory the destination satellite (where the kickstart repository still doesn't exists)
5. Import it using hammer

Actual results:

Task goes to pause/error state and logs this traceback:

    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 366, in publish
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 277, in populate
          setattr(self, f"{name}_repomdrecords", self.prepare_metadata_files(content, name))
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 89, in prepare_metadata_files
          content_artifact = repo_metadata_file.contentartifact_set.get()
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 408, in get
    description: ContentArtifact matching query does not exist.
  worker: "/pulp/api/v3/workers/111c5ae9-c32e-4c01-8f25-126f772c191d/"

Expected results:

Additional info:

Comment 2 Grant Gainey 2022-01-18 19:03:56 UTC
(Just an update so folk know this is still being investigated)

RHEL7-KS-REPO includes 3 productid files, one for the main repo and one for each of the addon-repos "inside" RHEL7-KS. AS an example from my test-machine, after syncing RHEL7 and RHEL7-KS here are the repometadata files:

pulp=> select rf.content_ptr_id, rf.relative_path, ca.pulp_id from rpm_repometadatafile rf LEFT JOIN core_contentartifact ca ON rf.content_ptr_id = ca.content_id;
            content_ptr_id            |                                     relative_path                                      |               pulp_id                
 430c399d-1381-476a-b162-fb3b10e8dfab | repodata/638a1ea9-66f5-4cd9-a1e8-505aa131e5b0                                          | 888a0329-7ce2-4eec-b5a7-2d91f6a56018
 8b1e01b1-9557-4934-9cb4-bd9cd4b96fe1 | repodata/54cd0bb6d2ad360b27c710b0437a4396b0f69fdd610a3df70f37628eed8039b5-productid.gz | b8bbfba4-8e37-4dd9-8302-30a8364d5108
 4a088a0b-c834-46b6-98ca-88f4b64ffd06 | repodata/716381b862bb7d32a7cc21e21e3c2ac167bd48332792da439b0806b97bd4cd92-productid.gz | 5212deb2-7d77-4954-9a00-5b2e24071245
 c78d5dac-4b77-4852-8504-4516f0b577c5 | repodata/de7d562330fef4aaed524325e7f2050f54897ff5129d92245fca2f927b8ac319-productid.gz | ec85cb78-3589-41ea-b34a-628748f3d0f9

and the repositories that hold them:
pulp=> select name, pulp_id from core_repository;
                                           name                                           |               pulp_id                
 rhel7                                                                                    | 026383f2-da36-4960-876b-0e4928d25e07
 rhel7_ks                                                                                 | 270b4eaf-8311-45df-83d7-bf94f2b413c1
 addons/ResilientStorage-f8e0ccff1237e1d7638db88802f090aedfcad5379390cd21d4bbe5a03bc2448f | bfd3a593-1b2e-4a6e-ad7a-3b4d25b7f2ff
 addons/HighAvailability-f8e0ccff1237e1d7638db88802f090aedfcad5379390cd21d4bbe5a03bc2448f | 363996b5-bbfa-4e32-a879-bffc7d02dae2
(4 rows)

The two addons' productid-repometadata-files aren't ending up in the rhel7-ks export (and hence not in the import, leading to the traceback). The only ContentArtifact with productid in the rel-path in my export is this:

    "artifact": "54cd0bb6d2ad360b27c710b0437a4396b0f69fdd610a3df70f37628eed8039b5",
    "content": "8b1e01b1-9557-4934-9cb4-bd9cd4b96fe1",
    "relative_path": "repodata/54cd0bb6d2ad360b27c710b0437a4396b0f69fdd610a3df70f37628eed8039b5-productid.gz"

So - the problem is happening at export-time. Investigation continues.

Comment 3 Grant Gainey 2022-01-18 19:09:35 UTC
Aaaand there it is.

Consider this code: https://github.com/pulp/pulpcore/blob/main/pulpcore/app/modelresource.py#L94-L97

When exporting, you only get resources associated with the repository you are exporting. In a kickstart repository, addons are each considered **their own repository**. Therefore, their content is not exported (including these repometadata files). However, the **content** of those addon directories **are** considered part of RHEL7-KS - so the content, including repometadata content, gets exported.

The ContentArtifact modelresource needs to know that some repositories "own" other repositories, and make sure to include such in the "parent" repo's export. Fun.

Comment 4 Grant Gainey 2022-01-18 19:13:41 UTC
3rd para in #c3 was unclear/incorrect, trying again:

When exporting, you only get resources associated with the repository you are exporting. In a kickstart repository, addons are each considered **their own repository**. Therefore, their ContentArtifact records ARE NOT exported.  However, the **content** of those addon directories (including these repometadata files) **are** considered part of RHEL7-KS - so the content, including repometadata content, gets exported.

Comment 5 Grant Gainey 2022-02-22 17:11:36 UTC
Fixed upstream, PR https://github.com/pulp/pulpcore/pull/2193 . Backported to core/3.14/15/16/17

Comment 8 Lai 2022-04-26 21:16:11 UTC
Steps to retest

1. spin up 2 satellites: 1 for connected and another for disconnected
2. on connected, import a manifest
3. in settings, change the download policy to 'immediate'
4. sync a rhel7 ks
5. create a cv and add repo
6. publish cv
7. export cv
8. on disconnected satellite, import manifest
9. set the cdn to air-gapped
10. in settings, change the download policy to 'immediate'
11. run import

Import should pass successfully

import does pass successfully

Verified on 6.11_017 with python38-pulpcore-3.16.7-1.el8pc.noarch.  Tested on both rhel7 and rhel8

Note You need to log in before you can comment on or make changes to this bug.