Bug 1946339 - ForeignKeyViolation Error with docker_meta_tags
Summary: ForeignKeyViolation Error with docker_meta_tags
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium vote
Target Milestone: 6.9.2
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-05 19:58 UTC by Lai
Modified: 2021-05-21 14:50 UTC (History)
7 users (show)

Fixed In Version: pulp-2to3-migration-0.11.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-20 18:05:30 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 8594 0 Normal CLOSED - CURRENTRELEASE Docker migration works improperly when some coruppted/missing content is being skipped 2021-05-04 17:28:12 UTC
Red Hat Product Errata RHBA-2021:2074 0 None None None 2021-05-20 18:05:40 UTC

Description Lai 2021-04-05 19:58:52 UTC
Description of problem:
Docker foreignKeyViolation after trying to run switchover from the dogfood clone.

Version-Release number of selected component (if applicable):
6.9.0_019.1


Steps to Reproduce:
1. Run migration on dogfood db
2. Once migration is done running, run switchover commands

Actual results:
PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags" violates foreign key constraint "fk_rails_af711a226d" on table "katello_repository_docker_meta_tags"
DETAIL:  Key (id)=(111) is still referenced from table "katello_repository_docker_meta_tags".

Expected results:
Switchover should be successful.

Additional info:

Comment 1 Justin Sherrill 2021-04-06 21:25:59 UTC
With some investigation, the underlying content in pulp3 was not migrated properly.  Ina is investigating.  It likely had to do with the cancelled migration that was restarted.

Comment 2 Justin Sherrill 2021-04-07 17:23:44 UTC
Ina did some more investigation and it appears its directly related to the dogfood server missing content on disk.  So to reproduce you'd likely need to rm -rf /var/lib/pulp/content/units/docker_*  before a migration

Comment 3 Justin Sherrill 2021-04-13 13:12:11 UTC
Created redmine issue https://projects.theforeman.org/issues/32306 from this bug

Comment 4 pulp-infra@redhat.com 2021-04-19 17:33:44 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2021-04-19 17:33:45 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2021-04-23 10:12:44 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2021-04-26 18:24:23 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 8 pulp-infra@redhat.com 2021-05-04 17:28:11 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 9 Ina Panova 2021-05-12 17:06:06 UTC
Steps to verify:

Scenario 1

1. sync in pulp2 hello-world from dockerhub
2. rf -rf /var/lib/pulp/content/units/docker_*
3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no content.
curl  https://<hostname>/pulp/api/v3/pulp2repositories/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key |python -m json.tool
`pulp3_repository_version` will point to a version 0 and it will be empty


Scenario 2
1. sync in pulp2 hello-world from dockerhub

2. Find pulp2 storage path for the manifest list tagged 'latest' and remove it from the FS
$ mongo
> use pulp_database
switched to db pulp_database

> db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"})
{ "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : {  }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : "units_docker_tag", "_content_type_id" : "docker_tag" }
> 
> db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1})
{ "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : "/var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" }

> ^C
bye

$ sudo rm  /var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 

3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no tag 'latest' 
http  'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container-push/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results

Comment 10 Ina Panova 2021-05-12 17:10:02 UTC
(In reply to Ina Panova from comment #9)
> Steps to verify:
> 
> Scenario 1
> 
> 1. sync in pulp2 hello-world from dockerhub
> 2. rf -rf /var/lib/pulp/content/units/docker_*
> 3. trigger migration with skip_corrupted=True
> 4. once migration is completed, check pulp3 migrated repo. It should have no
> content.
> curl  https://<hostname>/pulp/api/v3/pulp2repositories/ --cert
> /etc/pki/katello/certs/pulp-client.crt --key
> /etc/pki/katello/private/pulp-client.key |python -m json.tool
> `pulp3_repository_version` will point to a version 0 and it will be empty
> 
> 
> Scenario 2
> 1. sync in pulp2 hello-world from dockerhub
> 
> 2. Find pulp2 storage path for the manifest list tagged 'latest' and remove
> it from the FS
> $ mongo
> > use pulp_database
> switched to db pulp_database
> 
> > db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"})
> { "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : { 
> }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" :
> "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519",
> "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" :
> "units_docker_tag", "_content_type_id" : "docker_tag" }
> > 
> > db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1})
> { "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" :
> "/var/lib/pulp/content/units/docker_manifest_list/60/
> 0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:
> f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" }
> 
> > ^C
> bye
> 
> $ sudo rm 
> /var/lib/pulp/content/units/docker_manifest_list/60/
> 0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:
> f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 
> 
> 3. trigger migration with skip_corrupted=True
> 4. once migration is completed, check pulp3 migrated repo. It should have no
> tag 'latest' 

Correction:

 curl 'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results

Comment 11 Lai 2021-05-14 18:24:42 UTC
Steps to retest:

Here's the proper step to actually see the 'PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_docker_meta_tags"' error show up on a 6.9.0 and 6.9.1.

1. Sync 2 customer docker repos that has the same tags.  (I use `codefresh/cf-docker-puller` and `codefresh/cf-docker-builder` on quay)
2. Create a cv and add the 2 repos from step 1 and publish.
3. Run migration
4. Delete the cv created in step 2
5. Delete the customer repos and product made in step 1.
6. Recreate and resync the same docker repos in step 1.
7. Rerun migration
8. Run switchover command

Expected result:
Switchover command should run successfully without issues.

Actual Result:
Switchover command ran successfully without issues.

Note:  If you're on a 6.9.0 to 6.9.1, then you'll most likely encounter the `ForeignKeyVioloation` error.

=========================

Steps to verifying the migration should not contain tags referencing null manifest:

Scenario 1

1. sync in pulp2 hello-world from dockerhub
2. rm -rf /var/lib/pulp/content/units/docker_*
3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no content.
curl  https://<hostname>/pulp/api/v3/pulp2repositories/ --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key |python -m json.tool
`pulp3_repository_version` will point to a version 0 and it will be empty

Expected result:
`pulp3_repository_version` should have version 0 and should be empty

Actual Result:
"pulp3_repository_version": "/pulp/api/v3/repositories/container/container/1568e972-18dd-4a0a-883c-4e3f972aef98/versions/0/" and is empty



Scenario 2
1. sync in pulp2 hello-world from dockerhub

2. Find pulp2 storage path for the manifest list tagged 'latest' and remove it from the FS
$ mongo
> use pulp_database
switched to db pulp_database

> db.units_docker_tag.find({'name':'latest', 'repo_id':'hello-world', "manifest_type" : "list"})
{ "_id" : "6e3a961d-ccd5-45b5-8888-23ea5840843d", "pulp_user_metadata" : {  }, "_last_updated" : 1620057127, "name" : "latest", "manifest_digest" : "sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519", "repo_id" : "busy3", "schema_version" : 2, "manifest_type" : "list", "_ns" : "units_docker_tag", "_content_type_id" : "docker_tag" }
> 
> db.units_docker_manifest_list.find({'digest':'sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519'}, {'_storage_path':1})
{ "_id" : "5ee62ca8-1c41-4403-843d-d67eb628ec4f", "_storage_path" : "/var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519" }

> ^C
bye

$ sudo rm  /var/lib/pulp/content/units/docker_manifest_list/60/0fab2550fb0931069340b9d537fe04ece28fb0314391fc86555babd4e3ec65/sha256:f2266cbfc127c960fd30e76b7c792dc23b588c0db76233517e1891a4e357d519 

3. trigger migration with skip_corrupted=True
4. once migration is completed, check pulp3 migrated repo. It should have no tag 'latest' 
http  'https://<hostname>/pulp/api/v3/content/container/tags/?repository_version=/pulp/api/v3/repositories/container/container-push/<repo-version-uuid>/versions/1/&name=latest' -cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key should give 0 results


Expected:
Query should return no tag 'latest'

Actual:
{"count":0,"next":null,"previous":null,"results":[]}

verified on 6.9.2 snap 3 with python3-pulp-2to3-migration-0.11.1-1.el7pc.noarch

Comment 16 errata-xmlrpc 2021-05-20 18:05:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.2 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2074


Note You need to log in before you can comment on or make changes to this bug.