Bug 2243256

Summary: Unable to sync repository of type docker after running reclaim-space
Product: Red Hat Satellite Reporter: Eric Nothen <enothen>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Shweta Singh <shwsingh>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.13.3CC: dalley, dkliban, ggainey, ipanova, jrichards2, lmjachky, osousa, rchan, rlavi, sajha, shwsingh
Target Milestone: 6.15.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: pulp-container-2.14.8-1, pulp-container-2.17.0-1,pulp-container-2.16.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-04-23 17:15:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eric Nothen 2023-10-11 12:32:32 UTC
Description of problem:

I have a number of docker repositories with download policy on_demand, which have accumulated content (blobs) for months. As I don't need the older versions of the images anymore, I ran reclaim-space on these repositories, but I'm unable to sync them again afterwards.


Version-Release number of selected component (if applicable):
Satellite 6.13.3 on RHEL 8.8

How reproducible:
Always. Reproduced in 2 different environments.

Steps to Reproduce:

1. Create a product:
[root@satellite ~]# hammer product create --name "RHOSP16 Containers"
Product created.
[root@satellite ~]# 

2. Create a docker repository inside the product:
[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" --content-type docker --download-policy on_demand --upstream-username $username --upstream-password $password --url "https://registry.redhat.io" --docker-upstream-name rhosp-rhel8/openstack-aodh-base --name aodh-base
Repository created.
[root@satellite ~]# 

3. Verify sync completes successfully:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
Added Docker tags: 89
Total steps: 1377/1377
--------------------------------
Associating Content: 1121/1121
Downloading Artifacts: 166/166
Downloading tag list: 1/1
Processing Tags: 89/89
Un-Associating Content: 0/0
[root@satellite ~]# 

4. Reclaim space on the repository:
[root@satellite ~]# hammer repository reclaim-space --product "RHOSP16 Containers" --name aodh-base 
Could not reclaim the repository:
  undefined local variable or method `repositories' for #<Actions::Pulp3::Repository::ReclaimSpace:0x00007f0d552968f0>
  Did you mean?  repository_hrefs
[root@satellite ~]# 

5. Since the previous step didn't work using hammer, I went to the web interface, selected the repository and clicked on "Reclaim Space" in "Select Action". This task completed successfully

6. Attempting to sync afterwards:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
No content added.
Total steps: 1/90
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading tag list: 1/1
Processing Tags: 0/89
Error: 'NoneType' object has no attribute 'file'
[root@satellite ~]# 
[root@satellite ~]# hammer --output json repository info --product "RHOSP16 Containers" --name aodh-base | jq .Sync
{
  "Status": "Warning",
  "Last Sync Date": "1 minute"
}
[root@satellite ~]# 


Actual results: 
Sync fails

Expected results:
Sync works


Additional info:

Repository configuration:

[root@satellite ~]# hammer repository info --product "RHOSP16 Containers" --name aodh-base
Id:                          106
Name:                        aodh-base
Label:                       aodh-base
Description:                 
Organization:                LocalOrg
Red Hat Repository:          no
Content Type:                docker
Mirroring Policy:            Content Only
Url:                         https://registry.redhat.io
Publish Via HTTP:            yes
Published At:                satellite.local.lab/localorg-rhosp16_containers-aodh-base
Relative Path:               localorg-rhosp16_containers-aodh-base
Download Policy:             on_demand
Upstream Repository Name:    rhosp-rhel8/openstack-aodh-base
Container Image Tags Filter: 
Container Repository Name:   localorg-rhosp16_containers-aodh-base
HTTP Proxy:                  
    HTTP Proxy Policy: global_default_http_proxy
Product:                     
    Id:   303
    Name: RHOSP16 Containers
GPG Key:                     

Sync:                        
    Status:         Warning
    Last Sync Date: 1 minute
Created:                     2023/10/11 11:10:00
Updated:                     2023/10/11 11:11:57
Content Counts:              
    Container Image Manifest Lists: 72
    Container Image Manifests:      144
    Container Image Tags:           89

[root@satellite ~]# 


Traceback in messages:

[root@satellite ~]# grep 18522 /var/log/messages
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 11c9fdaf-ce0f-4567-ade5-d0712d19029b
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=aodh-base-1063035 remote=aodh-base-106854
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:46 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://registry.redhat.io/v2/'))
Oct 11 14:27:47 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulp_container.app.downloaders:INFO: Updating bearer token
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO: Task 11c9fdaf-ce0f-4567-ade5-d0712d19029b failed ('NoneType' object has no attribute 'file')
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]: pulp [0f325412-60ba-4c34-bd62-247885d366eb]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    result = func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/synchronize.py", line 41, in synchronize
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return dv.create()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    loop.run_until_complete(pipeline)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return future.result()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    await asyncio.gather(*futures)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    await self.run()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 152, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    listed_manifest = await listed_manifest_task
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 611, in _wait_for_one
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return f.result()  # May raise f.exception().
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 448, in create_listed_manifest
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    saved_artifact, content_data = await sync_to_async(_get_content_data_blocking)()
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 435, in __call__
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    ret = await asyncio.wait_for(future, timeout=None)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return await fut
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    result = self.fn(*self.args, **self.kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 476, in thread_handler
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    return func(*args, **kwargs)
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 444, in _get_content_data_blocking
Oct 11 14:27:49 satellite pulpcore-worker-1[18522]:    content_data = json.load(saved_artifact.file)

Comment 1 Eric Nothen 2023-10-11 12:40:43 UTC
The "Unauthorized" error in the 3rd line above suggests user/password could be wrong (which wouldn't explain how it sync'd the first time) so just in case, I'm creating a new repository with the same credentials:

[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" --content-type docker \
> --download-policy on_demand \
> --upstream-username $username \
> --upstream-password $password \
> --url "https://registry.redhat.io" \
> --docker-upstream-name rhosp-rhel8/openstack-aodh-evaluator \
> --name aodh-evaluator
Repository created.
[root@satellite ~]# 

And sync completes successfully:
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-evaluator
[.............................................................................................................................................................................................................................................] [100%]
Added Docker tags: 89
Total steps: 1539/1539
--------------------------------
Associating Content: 1287/1287
Downloading Artifacts: 162/162
Downloading tag list: 1/1
Processing Tags: 89/89
Un-Associating Content: 0/0
[root@satellite ~]# 
[root@satellite ~]# hammer --output json repository info --product "RHOSP16 Containers" --name aodh-evaluator | jq .Sync
{
  "Status": "Success",
  "Last Sync Date": "1 minute"
}
[root@satellite ~]#

Comment 2 Eric Nothen 2023-10-11 12:49:38 UTC
And finally, deleting and recreating the repository without any other pulp hack in the middle does not fix the issue:

[root@satellite ~]# hammer repository delete --product "RHOSP16 Containers" --name aodh-base
Repository deleted.
[root@satellite ~]# 
[root@satellite ~]# hammer repository create --product "RHOSP16 Containers" \
> --content-type docker \
> --download-policy on_demand \
> --upstream-username $username \
> --upstream-password $password \
> --url "https://registry.redhat.io" \
> --docker-upstream-name rhosp-rhel8/openstack-aodh-base \
> --name aodh-base
Repository created.
[root@satellite ~]# 
[root@satellite ~]# hammer repository sync --product "RHOSP16 Containers" --name aodh-base
[............................................................................................................................................................................................................] [100%]
No content added.
Total steps: 1/90
--------------------------------
Associating Content: 0/0
Downloading Artifacts: 0/0
Downloading tag list: 1/1
Processing Tags: 0/89
Error: 'NoneType' object has no attribute 'file'
[root@satellite ~]#

Comment 9 lmjachky 2023-10-31 11:02:18 UTC
The bugfix for pulp-container 2.14 has just been released:
https://docs.pulpproject.org/pulp_container/en/2.14/changes.html
https://pypi.org/project/pulp-container/2.14.8/

Comment 10 Robin Chan 2023-10-31 11:07:59 UTC
The Pulp upstream bug status is at closed. Updating the external tracker on this bug.

Comment 11 Robin Chan 2023-10-31 11:08:01 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 14 Shweta Singh 2024-01-31 13:11:57 UTC
Verified.

Version Tested: Satellite 6.15.0 Snap 8

Verification Steps: 
1. Create a custom product:
2. Create a docker repository inside the product.
3. Sync the repository and verify sync completes successfully.
4. Reclaim space on the repository.
6. Attempting to sync the repository again.

Result:
Repository gets synced successfully after reclaiming space.

Comment 17 errata-xmlrpc 2024-04-23 17:15:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2010