Created attachment 1800652 [details] error Description of problem: During the performance testing of pulp3, we are unable to sync docker images. Tried to sync 20 docker images and out of 20 , 7 got synced but 13 were failed with the following error : "undefined method `schema_version' for nil:NilClass." Version-Release number of selected component (if applicable): Red Hat Satellite (build: 6.10.0 Beta) How reproducible: Steps to Reproduce: 1. Sync docker 20 images using this playbook https://github.com/redhat-performance/satperf/blob/master/playbooks/tests/sync-docker.yaml Actual results: only 7 got synced, 13 were stopped Expected results: It should sync all the 20 docker images. Additional info:
You seem to sync repos from dockerhub, can you confirm? They have introduced rate pull limits.
Also, what is 6.10.0 Beta? To my knowledge, Beta is not out yet. Please share which pulp packages you have installed on your system. Thank you.
Imaan, please reply to the comments. Currently it looks like an issue of using dockerhub in testing.
Hi Tanya, I have tested on current snap 9 - Red Hat Satellite (build: 6.10.0 Beta)Version 6.10 © 2021 Red Hat Inc. I tried to sync 20 repositories of docker type, synced them in parallel, assigned each of them to one CV ,published them in parallel and promoted them in parallel. Every repo contains almost 4000 packages. Observation - out of 20, only 10 got successful and rest are failing with same error. while syncing where promote and published successfully completed. Let me check about docker hub. will update.
I logged into the instance and I could not find anything pulp related. All tasks were in the completed state as well repo versions were created, content was having properly associated artifacts to it. Logs also did not show anything. This is a katello error and it seems like it's being surfaced not for pulp reasons. @justin can you take a look please? Note: the repos were created and synced *not* from docker hub.
relevant traceback: NoMethodError undefined method `schema_version' for nil:NilClass --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:177:in `block in get_tag_table_values'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:176:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:176:in `map'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:176:in `get_tag_table_values'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:138:in `block in import_meta_tags'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:137:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_meta_tag.rb:137:in `import_meta_tags'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/docker_tag.rb:63:in `import_for_repository'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/models/katello/repository.rb:904:in `block (2 levels) in index_content'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1/app/lib/katello/logging.rb:8:in `time'"
Resyncing the repo seems to result in a successful sync, so this is likely a race condition.
Created redmine issue https://projects.theforeman.org/issues/33326 from this bug
I am proposing we remove this from a beta blocker. Syncing 10 very large docker repos (1000 tags each) at the same time i could not reproduce the issue, but i was able to reproduce syncing 20 very large docker repos at the same time.
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
moving to the pulp component as the underlying data issues i see are quite bad, and could be the cause of the katello error, although its not clear. I can re-test once we have a fix in pulp
Upstream bug assigned to jsherril
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Steps to retest 1. Create docker repo with 20+ images 2. sync repo 3. Check status of sync Expected: Sync should complete successfully Actual Sync completes successfully. I did a docker repo that has 20+ images and I was able to sync successfully without issues. Verified on 6.10 snap 20 with python3-pulp-container-2.8.1-0.1.el7pc.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4702