Bug 1982542
| Summary: | RefreshDistribution step fails with "publication":["Invalid hyperlink - Object does not exist."] | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Matthias Dellweg <mdellweg> |
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> |
| Status: | CLOSED ERRATA | QA Contact: | Stephen Wadeley <swadeley> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.9.0 | CC: | bbuckingham, dalley, ggainey, iballou, jjeffers, jsherril, osousa, pcreech, peter.vreman, rchan, ttereshc |
| Target Milestone: | 6.9.6 | Keywords: | Triaged |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | python-pulp_2to3_migration-0.11.4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-21 14:37:26 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1957813 | ||
It is only the optimized sync that fails. The Advanced sync and Verify Checksum of the repo works. It looks more on a mismatch between katello-pulp3 than an pulp3 internal issue There's actually two 400 errors in the sequence of events * Katello queries /distributions/... * Katello queries one specific distribution (fails 400 error) * Katello attempts to modify the same specific distribution (fails 400 error) It would be good to find out if this (https://bugzilla.redhat.com/show_bug.cgi?id=1983143) is related. The resulting Pulp API error is the same. Its actually slightly different: ["Invalid hyperlink - Incorrect URL match."]} vs ["Invalid hyperlink - Object does not exist."] I kinda wonder if something got orphan cleaned up by mistake @bbuckingham based on discussion with @ttereshc this appears to be relevant to pulp_2to3_migration, and not 6.10. James, Thanks for the feedback. We can align to 6.9.z, if that is the proper target. Has it been determined if the issue is within Katello or Pulp component? Brad, Current thinking seems to be pulp - there is a pulp issue https://pulp.plan.io/issues/9242 which may be related. It is not known for certain at this time. From the initial investigation and irc discussion:
- migration was interrupted and re-ran, no issues in logs, re-run finished successfully
- pulp2repositories mapping refers to at least one incomplete publication (which presumably came from the interruption of migration process)
- a repo version for the publication is complete
- there is a newer repository version with a complete publication
pulpcore=# select * from core_repositoryversion where repository_id = 'd00e56b6-78e1-40be-81c7-2bf4f5cd32d2';
pulp_id | pulp_created | pulp_last_updated | number | complete | base_version_id | repository_id
--------------------------------------+-------------------------------+-------------------------------+--------+----------+-----------------+--------------------------------------
b14ad863-7008-41b5-8bae-d476c9f25440 | 2021-08-16 15:55:15.415207+00 | 2021-08-16 15:55:15.41528+00 | 0 | t | | d00e56b6-78e1-40be-81c7-2bf4f5cd32d2
69651b08-0d0d-4b36-99f1-3ad3404b4779 | 2021-08-16 16:46:48.05274+00 | 2021-08-16 16:46:51.212159+00 | 1 | t | | d00e56b6-78e1-40be-81c7-2bf4f5cd32d2
6b8a9a14-6df4-40f7-a47d-31840e81c4cd | 2021-08-16 17:37:15.963792+00 | 2021-08-16 17:37:15.963808+00 | 2 | f | | d00e56b6-78e1-40be-81c7-2bf4f5cd32d2
(3 rows)
pulpcore=# select * from core_publication;
pulp_id | pulp_created | pulp_last_updated | pulp_type | complete | pass_through | repository_version_id
--------------------------------------+-------------------------------+-------------------------------+-----------+----------+--------------+--------------------------------------
c116583f-b11a-4620-be26-b2958507d124 | 2021-08-11 16:00:54.092536+00 | 2021-08-11 16:00:54.293128+00 | rpm.rpm | t | f | cec7b8a5-e054-43ed-a29d-82678c49a1bf
e0c323f8-a3b5-465a-bb3f-3c08cb813c0a | 2021-08-11 16:00:54.232569+00 | 2021-08-11 16:00:54.409348+00 | rpm.rpm | t | f | 691f9691-96aa-42a3-a050-6474568e9eef
2c10e3c6-0022-48b1-9ead-81120a3295b1 | 2021-08-11 16:14:41.825803+00 | 2021-08-11 16:14:42.084446+00 | rpm.rpm | t | f | d9155ec7-f874-452d-9b4b-5d20af5d7e61
02219c49-bd3d-42c9-883a-4bfd743f9818 | 2021-08-12 19:37:38.210234+00 | 2021-08-12 19:37:38.462139+00 | rpm.rpm | t | f | 8d59aa81-4c45-4c2c-a392-e1f5a58c7a53
56516077-2459-43ee-bad8-983605482f25 | 2021-08-12 20:08:23.293607+00 | 2021-08-12 20:08:23.696868+00 | rpm.rpm | t | f | 18172bbd-30fc-4dd9-8156-503efbfb652e
f42b6921-d8a4-44aa-af7f-a7526272d933 | 2021-08-16 15:59:15.736648+00 | 2021-08-16 16:00:54.445885+00 | rpm.rpm | t | f | 72b7ca49-34bb-430c-8e85-1caba58ee267
41877d72-ced3-4034-bafc-2c9f81686b1c | 2021-08-16 16:46:47.393314+00 | 2021-08-16 16:46:47.571879+00 | rpm.rpm | t | f | ae8ecbc2-5acb-4370-bebb-12a9b7e7ce0b
657e601a-3386-403c-810b-2f4409865810 | 2021-08-16 16:46:51.345391+00 | 2021-08-16 16:46:51.345414+00 | rpm.rpm | f | f | 69651b08-0d0d-4b36-99f1-3ad3404b4779
(8 rows)
pulpcore=# select * from core_publication where repository_version_id = '69651b08-0d0d-4b36-99f1-3ad3404b4779';
pulp_id | pulp_created | pulp_last_updated | pulp_type | complete | pass_through | repository_version_id
--------------------------------------+-------------------------------+-------------------------------+-----------+----------+--------------+--------------------------------------
657e601a-3386-403c-810b-2f4409865810 | 2021-08-16 16:46:51.345391+00 | 2021-08-16 16:46:51.345414+00 | rpm.rpm | f | f | 69651b08-0d0d-4b36-99f1-3ad3404b4779
I'd expect Pulp to either cleanup the incomplete publication at the beginning of a re-run, or ignore it and it won't be exposed to a user (it will be eventually deleted with the removal of repository version)
Maybe it helps as input for analysis, i left it already as comment in the HTB group last month The issue for the 3 repos 'self-repaired' and also the optimized sync started to work normally again. It was sometime (5-20 days) after the pulp2-3 migration and switchover was done. It might be The Pulp upstream bug status is at NEW. Updating the external tracker on this bug. The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug. The Pulp upstream bug status is at NEW. Updating the external tracker on this bug. The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug. Step I used to reproduce this on katello 3.18 dev environment: 1. Started a migration (foreman-maintain content prepare) 2. Cancelled the migration with ctrl-c (sigint) 3. Restarted the migration, observed warning about missing or corrupted content 4. ran corrupted content approval task 5. Restarted migration 6. Run content switchover task. 6. After migration, attempt re-sync of migrated repository. The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST. The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. Can I get some sample repos that is hitting this issue for testing? (In reply to Lai from comment #25) > Can I get some sample repos that is hitting this issue for testing? IIRC I used an RHEL 7 epel repository (7k+ packages), because I needed a sufficient window of time to cancel the migration, etc. I don't think there was anything special about the repository that would recreated the problem, as long as the procedure is like described. Hello Bug 2004248 - AVC denials on new instance of 6.9.6 snap 2 Hello Bug 2006307 - Pulp2-Pulp3 migration failed due to mongodb oom errors Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Satellite 6.9.6 Async Bug Fix Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3628 |
Description of problem: Synchronization of some repositories fails after being migrated to pulp3. The RefreshDistribution step errors with a bad request to pulp, complaining the publication does not exist. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Error in dynflow: ``` PulpRpmClient::ApiError Error message: the server returns an error HTTP status code: 400 Response headers: {"Date"=>"Wed, 14 Jul 2021 08:48:03 GMT", "Server"=>"gunicorn/20.0.4", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"62", "Via"=>"1.1 <redacted>.com", "Connection"=>"close"} Response body: {"publication":["Invalid hyperlink - Object does not exist."]} ``` Pulp log message: ``` pulp: django.request:WARNING: Bad Request: /pulp/api/v3/distributions/rpm/rpm/91b708b0-8a7b-41e1-a27f-13e591c8a474/ pulpcore-api[1593]: pulp [-]: - - [14/Jul/2021:08:48:03 +0000] "PATCH /pulp/api/v3/distributions/rpm/rpm/91b708b0-8a7b-41e1-a27f-13e591c8a474/ HTTP/1.1" 400 62 "-" "OpenAPI-Generator/3.10.0/ruby" ``` Expected results: Additional info: