Bug 1946339

With some investigation, the underlying content in pulp3 was not migrated properly.  Ina is investigating.  It likely had to do with the cancelled migration that was restarted.

Ina did some more investigation and it appears its directly related to the dogfood server missing content on disk.  So to reproduce you'd likely need to rm -rf /var/lib/pulp/content/units/docker_*  before a migration

Created redmine issue https://projects.theforeman.org/issues/32306 from this bug

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Steps to verify:

Scenario 1

1. sync in pulp2 hello-world from dockerhub
2. rf -rf /var/lib/pulp/content/units/docker_*
3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no content.
curl  https://<hostname>/pulp/api/v3/pulp2repositories/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key |python -m json.tool
`pulp3_repository_version` will point to a version 0 and it will be empty


Scenario 2
1. sync in pulp2 hello-world from dockerhub

2. Find pulp2 storage path for the manifest list tagged 'latest' and remove it from the FS
$ mongo
> use pulp_database
switched to db pulp_database

> db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"})
{ "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : {  }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : "units_docker_tag", "_content_type_id" : "docker_tag" }
> 
> db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1})
{ "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : "/var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" }

> ^C
bye

$ sudo rm  /var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 

3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no tag 'latest' 
http  'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container-push/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results

(In reply to Ina Panova from comment #9)
> Steps to verify:
> 
> Scenario 1
> 
> 1. sync in pulp2 hello-world from dockerhub
> 2. rf -rf /var/lib/pulp/content/units/docker_*
> 3. trigger migration with skip_corrupted=True
> 4. once migration is completed, check pulp3 migrated repo. It should have no
> content.
> curl  https://<hostname>/pulp/api/v3/pulp2repositories/ --cert
> /etc/pki/katello/certs/pulp-client.crt --key
> /etc/pki/katello/private/pulp-client.key |python -m json.tool
> `pulp3_repository_version` will point to a version 0 and it will be empty
> 
> 
> Scenario 2
> 1. sync in pulp2 hello-world from dockerhub
> 
> 2. Find pulp2 storage path for the manifest list tagged 'latest' and remove
> it from the FS
> $ mongo
> > use pulp_database
> switched to db pulp_database
> 
> > db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"})
> { "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : { 
> }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" :
> "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519",
> "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" :
> "units_docker_tag", "_content_type_id" : "docker_tag" }
> > 
> > db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1})
> { "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" :
> "/var/lib/pulp/content/units/docker_manifest_list/60/
> 0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:
> f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" }
> 
> > ^C
> bye
> 
> $ sudo rm 
> /var/lib/pulp/content/units/docker_manifest_list/60/
> 0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:
> f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 
> 
> 3. trigger migration with skip_corrupted=True
> 4. once migration is completed, check pulp3 migrated repo. It should have no
> tag 'latest' 

Correction:

 curl 'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results

Steps to retest:

Here's the proper step to actually see the 'PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags"' error show up on a 6.9.0 and 6.9.1.

1. Sync 2 customer docker repos that has the same tags.  (I use `codefresh/cf-docker-puller` and `codefresh/cf-docker-builder` on quay)
2. Create a cv and add the 2 repos from step 1 and publish.
3. Run migration
4. Delete the cv created in step 2
5. Delete the customer repos and product made in step 1.
6. Recreate and resync the same docker repos in step 1.
7. Rerun migration
8. Run switchover command

Expected result:
Switchover command should run successfully without issues.

Actual Result:
Switchover command ran successfully without issues.

Note:  If you're on a 6.9.0 to 6.9.1, then you'll most likely encounter the `ForeignKeyVioloation` error.

=========================

Steps to verifying the migration should not contain tags referencing null manifest:

Scenario 1

1. sync in pulp2 hello-world from dockerhub
2. rm -rf /var/lib/pulp/content/units/docker_*
3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no content.
curl  https://<hostname>/pulp/api/v3/pulp2repositories/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key |python -m json.tool
`pulp3_repository_version` will point to a version 0 and it will be empty

Expected result:
`pulp3_repository_version` should have version 0 and should be empty

Actual Result:
"pulp3_repository_version": "/pulp/api/v3/repositories/container/container/1568e972-18dd-4a0a-883c-4e3f972aef98/versions/0/" and is empty



Scenario 2
1. sync in pulp2 hello-world from dockerhub

2. Find pulp2 storage path for the manifest list tagged 'latest' and remove it from the FS
$ mongo
> use pulp_database
switched to db pulp_database

> db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"})
{ "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : {  }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : "units_docker_tag", "_content_type_id" : "docker_tag" }
> 
> db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1})
{ "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : "/var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" }

> ^C
bye

$ sudo rm  /var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 

3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no tag 'latest' 
http  'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container-push/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results


Expected:
Query should return no tag 'latest'

Actual:
{"count":0,"next":null,"previous":null,"results":[]}

verified on 6.9.2 snap 3 with python3-pulp-2to3-migration-0.11.1-1.el7pc.noarch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.2 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2074