Description of problem: Docker foreignKeyViolation after trying to run switchover from the dogfood clone. Version-Release number of selected component (if applicable): 6.9.0_019.1 Steps to Reproduce: 1. Run migration on dogfood db 2. Once migration is done running, run switchover commands Actual results: PG::ForeignKeyViolation: ERROR: update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags" DETAIL: Key (id)=(111) is still referenced from table "katello_repository_docker_meta_tags". Expected results: Switchover should be successful. Additional info:
With some investigation, the underlying content in pulp3 was not migrated properly. Ina is investigating. It likely had to do with the cancelled migration that was restarted.
Ina did some more investigation and it appears its directly related to the dogfood server missing content on disk. So to reproduce you'd likely need to rm -rf /var/lib/pulp/content/units/docker_* before a migration
Created redmine issue https://projects.theforeman.org/issues/32306 from this bug
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Steps to verify: Scenario 1 1. sync in pulp2 hello-world from dockerhub 2. rf -rf /var/lib/pulp/content/units/docker_* 3. trigger migration with skip_corrupted=True 4. once migration is completed, check pulp3 migrated repo. It should have no content. curl https://<hostname>/pulp/api/v3/pulp2repositories/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key |python -m json.tool `pulp3_repository_version` will point to a version 0 and it will be empty Scenario 2 1. sync in pulp2 hello-world from dockerhub 2. Find pulp2 storage path for the manifest list tagged 'latest' and remove it from the FS $ mongo > use pulp_database switched to db pulp_database > db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"}) { "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : { }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : "units_docker_tag", "_content_type_id" : "docker_tag" } > > db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1}) { "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : "/var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" } > ^C bye $ sudo rm /var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 3. trigger migration with skip_corrupted=True 4. once migration is completed, check pulp3 migrated repo. It should have no tag 'latest' http 'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container-push/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results
(In reply to Ina Panova from comment #9) > Steps to verify: > > Scenario 1 > > 1. sync in pulp2 hello-world from dockerhub > 2. rf -rf /var/lib/pulp/content/units/docker_* > 3. trigger migration with skip_corrupted=True > 4. once migration is completed, check pulp3 migrated repo. It should have no > content. > curl https://<hostname>/pulp/api/v3/pulp2repositories/ --cert > /etc/pki/katello/certs/pulp-client.crt --key > /etc/pki/katello/private/pulp-client.key |python -m json.tool > `pulp3_repository_version` will point to a version 0 and it will be empty > > > Scenario 2 > 1. sync in pulp2 hello-world from dockerhub > > 2. Find pulp2 storage path for the manifest list tagged 'latest' and remove > it from the FS > $ mongo > > use pulp_database > switched to db pulp_database > > > db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"}) > { "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : { > }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : > "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", > "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : > "units_docker_tag", "_content_type_id" : "docker_tag" } > > > > db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1}) > { "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : > "/var/lib/pulp/content/units/docker_manifest_list/60/ > 0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256: > f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" } > > > ^C > bye > > $ sudo rm > /var/lib/pulp/content/units/docker_manifest_list/60/ > 0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256: > f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 > > 3. trigger migration with skip_corrupted=True > 4. once migration is completed, check pulp3 migrated repo. It should have no > tag 'latest' Correction: curl 'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results
Steps to retest: Here's the proper step to actually see the 'PG::ForeignKeyViolation: ERROR: update or delete on table "katello_docker_meta_tags"' error show up on a 6.9.0 and 6.9.1. 1. Sync 2 customer docker repos that has the same tags. (I use `codefresh/cf-docker-puller` and `codefresh/cf-docker-builder` on quay) 2. Create a cv and add the 2 repos from step 1 and publish. 3. Run migration 4. Delete the cv created in step 2 5. Delete the customer repos and product made in step 1. 6. Recreate and resync the same docker repos in step 1. 7. Rerun migration 8. Run switchover command Expected result: Switchover command should run successfully without issues. Actual Result: Switchover command ran successfully without issues. Note: If you're on a 6.9.0 to 6.9.1, then you'll most likely encounter the `ForeignKeyVioloation` error. ========================= Steps to verifying the migration should not contain tags referencing null manifest: Scenario 1 1. sync in pulp2 hello-world from dockerhub 2. rm -rf /var/lib/pulp/content/units/docker_* 3. trigger migration with skip_corrupted=True 4. once migration is completed, check pulp3 migrated repo. It should have no content. curl https://<hostname>/pulp/api/v3/pulp2repositories/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key |python -m json.tool `pulp3_repository_version` will point to a version 0 and it will be empty Expected result: `pulp3_repository_version` should have version 0 and should be empty Actual Result: "pulp3_repository_version": "/pulp/api/v3/repositories/container/container/1568e972-18dd-4a0a-883c-4e3f972aef98/versions/0/" and is empty Scenario 2 1. sync in pulp2 hello-world from dockerhub 2. Find pulp2 storage path for the manifest list tagged 'latest' and remove it from the FS $ mongo > use pulp_database switched to db pulp_database > db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"}) { "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : { }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : "units_docker_tag", "_content_type_id" : "docker_tag" } > > db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1}) { "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : "/var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" } > ^C bye $ sudo rm /var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 3. trigger migration with skip_corrupted=True 4. once migration is completed, check pulp3 migrated repo. It should have no tag 'latest' http 'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container-push/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results Expected: Query should return no tag 'latest' Actual: {"count":0,"next":null,"previous":null,"results":[]} verified on 6.9.2 snap 3 with python3-pulp-2to3-migration-0.11.1-1.el7pc.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Satellite 6.9.2 Async Bug Fix Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2074