Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2243256 - Unable to sync repository of type docker after running reclaim-space
Summary: Unable to sync repository of type docker after running reclaim-space
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.13.3
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: 6.15.0
Assignee: satellite6-bugs
QA Contact: Shweta Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-10-11 12:32 UTC by Eric Nothen
Modified: 2024-04-23 17:15 UTC (History)
11 users (show)

Fixed In Version: pulp-container-2.14.8-1, pulp-container-2.17.0-1,pulp-container-2.16.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-04-23 17:15:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github pulp pulp_container issues 1400 0 None closed Unable to sync repository after running reclaim-space 2023-10-31 11:07:58 UTC
Red Hat Issue Tracker SAT-20711 0 None None None 2023-10-11 12:34:31 UTC
Red Hat Product Errata RHSA-2024:2010 0 None None None 2024-04-23 17:15:06 UTC

Description Eric Nothen 2023-10-11 12:32:32 UTC 
Description of problem:

I have a number of docker repositories with download policy on_demand, which have accumulated content (blobs) for months. As I don't need the older versions of the images anymore, I ran reclaim-space on these repositories, but I'm unable to sync them again afterwards.


Version-Release number of selected component (if applicable):
Satellite 6.13.3 on RHEL 8.8

How reproducible:
Always. Reproduced in 2 different environments.

Steps to Reproduce:

1. Create a product:
[root@satellite ~]# hammer product create --name "RHOSP16 Containers"
Product created.
[root@satellite ~]# 

2. Create a docker repository inside the product:
[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" --content-type docker --download-policy on_demand --upstream-username $username --upstream-password $password --url "https://registry.redhat.io" --docker-upstream-name rhosp-rhel8/openstack-aodh-base --name aodh-base
Repository created.
[root@satellite ~]# 

3. Verify sync completes successfully:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
Added Docker tags: 89
Total steps: 1377/1377
--------------------------------
Associating Content: 1121/1121
Downloading Artifacts: 166/166
Downloading tag list: 1/1
Processing Tags: 89/89
Un-Associating Content: 0/0
[root@satellite ~]# 

4. Reclaim space on the repository:
[root@satellite ~]# hammer repository reclaim-space --product "RHOSP16 Containers" --name aodh-base 
Could not reclaim the repository:
  undefined local variable or method `repositories' for #<Actions::Pulp3::Repository::ReclaimSpace:0x00007f0d552968f0>
  Did you mean?  repository_hrefs
[root@satellite ~]# 

5. Since the previous step didn't work using hammer, I went to the web interface, selected the repository and clicked on "Reclaim Space" in "Select Action". This task completed successfully

6. Attempting to sync afterwards:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
No content added.
Total steps: 1/90
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading tag list: 1/1
Processing Tags: 0/89
Error: 'NoneType' object has no attribute 'file'
[root@satellite ~]# 
[root@satellite ~]# hammer --output json repository info --product "RHOSP16 Containers" --name aodh-base | jq .Sync
{
  "Status": "Warning",
  "Last Sync Date": "1 minute"
}
[root@satellite ~]# 


Actual results: 
Sync fails

Expected results:
Sync works


Additional info:

Repository configuration:

[root@satellite ~]# hammer repository info --product "RHOSP16 Containers" --name aodh-base
Id:                          106
Name:                        aodh-base
Label:                       aodh-base
Description:                 
Organization:                LocalOrg
Red Hat Repository:          no
Content Type:                docker
Mirroring Policy:            Content Only
Url:                         https://registry.redhat.io
Publish Via HTTP:            yes
Published At:                satellite.local.lab/localorg-rhosp16_containers-aodh-base
Relative Path:               localorg-rhosp16_containers-aodh-base
Download Policy:             on_demand
Upstream Repository Name:    rhosp-rhel8/openstack-aodh-base
Container Image Tags Filter: 
Container Repository Name:   localorg-rhosp16_containers-aodh-base
HTTP Proxy:                  
    HTTP Proxy Policy: global_default_http_proxy
Product:                     
    Id:   303
    Name: RHOSP16 Containers
GPG Key:                     

Sync:                        
    Status:         Warning
    Last Sync Date: 1 minute
Created:                     2023/10/11 11:10:00
Updated:                     2023/10/11 11:11:57
Content Counts:              
    Container Image Manifest Lists: 72
    Container Image Manifests:      144
    Container Image Tags:           89

[root@satellite ~]# 


Traceback in messages:

[root@satellite ~]# grep 18522 /var/log/messages
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 11c9fdaf-ce0f-4567-ade5-d0712d19029b
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=aodh-base-1063035 remote=aodh-base-106854
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:47 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.downloaders:INFO: Updating bearer token
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Task 11c9fdaf-ce0f-4567-ade5-d0712d19029b failed ('NoneType' object has no attribute 'file')
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    result = func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/synchronize.py", line 41, in synchronize
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return dv.create()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    loop.run_until_complete(pipeline)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return future.result()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    await asyncio.gather(*futures)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    await self.run()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 152, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    listed_manifest = await listed_manifest_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 611, in _wait_for_one
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return f.result()  # May raise f.exception().
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 448, in create_listed_manifest
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    saved_artifact, content_data = await sync_to_async(_get_content_data_blocking)()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 435, in __call__
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    ret = await asyncio.wait_for(future, timeout=None)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return await fut
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    result = self.fn(*self.args, **self.kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 476, in thread_handler
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 444, in _get_content_data_blocking
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    content_data = json.load(saved_artifact.file)
Comment 1 Eric Nothen 2023-10-11 12:40:43 UTC 
The "Unauthorized" error in the 3rd line above suggests user/password could be wrong (which wouldn't explain how it sync'd the first time) so just in case, I'm creating a new repository with the same credentials:

[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" --content-type docker \
> --download-policy on_demand \
> --upstream-username $username \
> --upstream-password $password \
> --url "https://registry.redhat.io" \
> --docker-upstream-name rhosp-rhel8/openstack-aodh-evaluator \
> --name aodh-evaluator
Repository created.
[root@satellite ~]# 

And sync completes successfully:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-evaluator
[.............................................................................................................................................................................................................................................] [100%]
Added Docker tags: 89
Total steps: 1539/1539
--------------------------------
Associating Content: 1287/1287
Downloading Artifacts: 162/162
Downloading tag list: 1/1
Processing Tags: 89/89
Un-Associating Content: 0/0
[root@satellite ~]# 
[root@satellite ~]# hammer --output json repository info --product "RHOSP16 Containers" --name aodh-evaluator | jq .Sync
{
  "Status": "Success",
  "Last Sync Date": "1 minute"
}
[root@satellite ~]#
Comment 2 Eric Nothen 2023-10-11 12:49:38 UTC 
And finally, deleting and recreating the repository without any other pulp hack in the middle does not fix the issue:

[root@satellite ~]# hammer repository delete --product "RHOSP16 Containers" --name aodh-base
Repository deleted.
[root@satellite ~]# 
[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" \
> --content-type docker \
> --download-policy on_demand \
> --upstream-username $username \
> --upstream-password $password \
> --url "https://registry.redhat.io" \
> --docker-upstream-name rhosp-rhel8/openstack-aodh-base \
> --name aodh-base
Repository created.
[root@satellite ~]# 
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
No content added.
Total steps: 1/90
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading tag list: 1/1
Processing Tags: 0/89
Error: 'NoneType' object has no attribute 'file'
[root@satellite ~]#
Comment 9 lmjachky 2023-10-31 11:02:18 UTC 
The bugfix for pulp-container 2.14 has just been released:
https://docs.pulpproject.org/pulp_container/en/2.14/changes.html
https://pypi.org/project/pulp-container/2.14.8/
Comment 10 Robin Chan 2023-10-31 11:07:59 UTC 
The Pulp upstream bug status is at closed. Updating the external tracker on this bug.
Comment 11 Robin Chan 2023-10-31 11:08:01 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 14 Shweta Singh 2024-01-31 13:11:57 UTC 
Verified.

Version Tested: Satellite 6.15.0 Snap 8

Verification Steps: 
1. Create a custom product:
2. Create a docker repository inside the product.
3. Sync the repository and verify sync completes successfully.
4. Reclaim space on the repository.
6. Attempting to sync the repository again.

Result:
Repository gets synced successfully after reclaiming space.
Comment 17 errata-xmlrpc 2024-04-23 17:15:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2010
Note You need to log in before you can comment on or make changes to this bug.