2107701 – [Pulp 3] If a modulemd metadata artifact is missing from the filesystem but has an artifact_id associated with it in database, "Verify Content Checksum" cannot fix this problem

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2107701 - [Pulp 3] If a modulemd metadata artifact is missing from the filesystem but has an artifact_id associated with it in database, "Verify Content Checksum" cannot fix this problem

Summary: [Pulp 3] If a modulemd metadata artifact is missing from the filesystem but h...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Pulp
Sub Component:
Version:	6.11.0
Hardware:	All
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	6.12.0
Assignee:	satellite6-bugs
QA Contact:	Vladimír Sedmík
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-07-15 16:59 UTC by Sayan Das
Modified:	2024-01-31 14:58 UTC (History)
CC List:	5 users (show)
Fixed In Version:	pulp_rpm 3.18.2
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-11-16 13:34:39 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	pulp pulp_rpm issues 2621	None	closed	Consider storing modulemd data in a (hidden) textfield rather than as individual artifacts	2022-09-09 13:29:59 UTC
Github	pulp pulp_rpm issues 2735	None	closed	With a combination of failed and successful syncs, it is possible to have modulemd content with no artifact associated	2022-10-13 16:30:37 UTC
Red Hat Issue Tracker	SAT-12380	None	None	None	2022-08-23 00:26:45 UTC
Red Hat Knowledge Base (Solution)	6961905	None	None	None	2022-07-18 12:43:19 UTC
Red Hat Product Errata	RHSA-2022:8506	None	None	None	2022-11-16 13:34:54 UTC

Description Sayan Das 2022-07-15 16:59:07 UTC

Description of problem:

With a combination of multiple failed and successful sync attempts on a repo like "Appstream 8", It may happen that, One or more artifacts related to modulemd metdata is physically missing from the filesystem but if we check the pulpcore database, then the same artifact is having an artifact_id associated with it which means pulp thinks the file is alredy created\downloaded on the filesystem.

This can happen for multiple modulemd files and an attempt of exporting the repository will fail in these circumstances:

# hammer content-export complete repository --organization="XXXX" --id='791'
[...............................................................................................................................................................................................] [100%]
Error: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1'
undefined method `first' for nil:NilClass

The repair feature of pulp, Integrated with "Validate content checksum" option of Katello, shows that it was able to find and fix a missing file from the concerned "Appstream 8" repo but actually , that modulemd artifact is still missing from the filesystem.


Version-Release number of selected component (if applicable):

Satellite 6.10 and 6.11
Pulp 3


How reproducible:

Always


Steps to Reproduce:
1. Install Satellite 6.11 on RHEL 8 | or 6.10 on RHEl 7
2. Enable and sync the "Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8" repo.
3. Check for the availability of this file in filesystem i.e. /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
4. Check for the presence of this artifact artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1 in pulpcore db
5. Remove that file from filesystem.
6. "Complete Sync" the repo and check if the file came back.
7. Run a "Verify content checksum" against teh same repo and check the results.

Actual results:

On FS:

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  File: /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  Size: 2480      	Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 163610175   Links: 1
Access: (0644/-rw-r--r--)  Uid: (  988/    pulp)   Gid: (  985/    pulp)
Context: system_u:object_r:pulpcore_var_lib_t:s0
Access: 2022-07-15 02:22:18.547076698 +0530
Modify: 2022-06-08 06:32:07.524593523 +0530
Change: 2022-06-08 06:36:17.184745061 +0530
 Birth: 2022-06-08 06:32:07.524593523 +0530
 
# head /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
---
document: modulemd
version: 2
data:
  name: perl-YAML
  stream: "1.24"
  version: 820181214175558
  context: 8652dbeb
  arch: x86_64
  summary: Perl parser for YAML


in DB:

# echo "select ca.pulp_id,cca.artifact_id,ca.file,cca.relative_path from core_artifact ca LEFT JOIN core_contentartifact cca on cca.artifact_id = ca.pulp_id where ca.file = 'artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1';"   | su - postgres -c "psql -x pulpcore"
-[ RECORD 1 ]-+---------------------------------------------------------------------------
pulp_id       | 17e28edc-0195-4115-922c-863f40e7ef62
artifact_id   | 17e28edc-0195-4115-922c-863f40e7ef62
file          | artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
relative_path | perl-YAML1.248201812141755588652dbebx86_64snippet

# echo "select pulp_id,pulp_created,pulp_last_updated,file,sha256 from core_artifact where file = 'artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1';"    | su - postgres -c "psql -x pulpcore"
-[ RECORD 1 ]-----+---------------------------------------------------------------------------
pulp_id           | 17e28edc-0195-4115-922c-863f40e7ef62
pulp_created      | 2022-06-08 06:36:17.185196+05:30
pulp_last_updated | 2022-06-08 06:36:17.185208+05:30
file              | artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
sha256            | a38b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1


# echo "select * from core_contentartifact where artifact_id = '17e28edc-0195-4115-922c-863f40e7ef62';"    | su - postgres -c "psql -x pulpcore" 
-[ RECORD 1 ]-----+--------------------------------------------------
pulp_id           | 3c8a3f07-703e-4c82-835f-a3292dc83bed
pulp_created      | 2022-02-19 16:29:04.054172+05:30
pulp_last_updated | 2022-02-19 16:29:04.054176+05:30
relative_path     | perl-YAML1.248201812141755588652dbebx86_64snippet
artifact_id       | 17e28edc-0195-4115-922c-863f40e7ef62
content_id        | 05c92fad-816c-425a-8dc5-a1e40c947c87

Delete:

# rm /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
rm: remove regular file '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1'? y

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
stat: cannot statx '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1': No such file or directory

Simple Sync: No luck

Verify Content Sync: Even though it shows the missing unit has been fixed, still no luck. The file is still absent.

Products -> Red Hat Enterprise Linux for x86_64 -> Repositories -> Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8 -> Select Action -> Verify Content Checksum


Jul 15 21:01:21 satellite611 pulpcore-api[124567]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]:  - - [15/Jul/2022:15:31:21 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/3d8caef4-e94a-4092-934d-c79937a7adf9/versions/40/repair/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.17.4/ruby"
Jul 15 21:01:21 satellite611 pulpcore-api[124567]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]:  - - [15/Jul/2022:15:31:21 +0000] "GET /pulp/api/v3/tasks/6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5/ HTTP/1.1" 200 572 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 15 21:01:21 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5
..
Jul 15 21:01:21 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.app.tasks.repository:INFO: Repairing version 40 of repository 'Red_Hat_Enterprise_Linux_8_for_x86_64_-_AppStream_RPMs_8-176782'
..
Jul 15 21:01:22 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.app.tasks.repository:WARNING: Missing file for <Artifact: pk=17e28edc-0195-4115-922c-863f40e7ef62>
..
Jul 15 21:01:23 satellite611 pulpcore-api[161362]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]:  - - [15/Jul/2022:15:31:23 +0000] "GET /pulp/api/v3/tasks/6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5/ HTTP/1.1" 200 946 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 15 21:01:23 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5



# curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://`hostname -f`/pulp/api/v3/tasks/6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5/ | jq .progress_reports
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   979  100   979    0     0  16316      0 --:--:-- --:--:-- --:--:-- 16593
[
  {
    "message": "Identify missing units",
    "code": "repair.missing",
    "state": "completed",
    "total": null,
    "done": 1,
    "suffix": null
  },
  {
    "message": "Identify corrupted units",
    "code": "repair.corrupted",
    "state": "completed",
    "total": null,
    "done": 0,
    "suffix": null
  },
  {
    "message": "Repair corrupted units",
    "code": "repair.repaired",
    "state": "completed",
    "total": null,
    "done": 0,
    "suffix": null
  }
]

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
stat: cannot statx '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1': No such file or directory


Republished Repository metadata: But that also did not helped.

Complete Sync done: That also did not helped. 


Expected results:

If modulemd metadata files are supposed to be downloaded from remote only, then repari action via "Verify Content Checksum" should work.

If modulemd metadata files are created by pulp similar to repository metadata files, then a "Republish" action on the repo via API or "Complete Sync" should be able to recreate the file.


Additional info:

The only way i can fix it is :

** nullify the artifact_id from pulpcore db core_contentartifact table and delete the record of that artifact from core_artifact

# su - postgres -c "psql pulpcore"
psql (12.11)
Type "help" for help.

pulpcore=# UPDATE core_contentartifact SET artifact_id = NULL WHERE relative_path = 'perl-YAML1.248201812141755588652dbebx86_64snippet'
pulpcore-# ;
UPDATE 1
pulpcore=# delete from  core_artifact where pulp_id = '17e28edc-0195-4115-922c-863f40e7ef62';
DELETE 1
pulpcore=# exit


** Then "Complete Sync" the repo. 

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  File: /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  Size: 2480      	Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 306281556   Links: 1
Access: (0644/-rw-r--r--)  Uid: (  988/    pulp)   Gid: (  985/    pulp)
Context: system_u:object_r:pulpcore_var_lib_t:s0
Access: 2022-07-15 21:07:12.437638569 +0530
Modify: 2022-07-15 21:07:12.436638480 +0530
Change: 2022-07-15 21:10:36.189788367 +0530
 Birth: 2022-07-15 21:07:12.436638480 +0530


# sha256sum /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
a38b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1  /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1


# echo "select ca.pulp_id,cca.artifact_id,ca.file,cca.relative_path from core_artifact ca LEFT JOIN core_contentartifact cca on cca.artifact_id = ca.pulp_id where ca.file = 'artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1';"   | su - postgres -c "psql -x pulpcore"
-[ RECORD 1 ]-+---------------------------------------------------------------------------
pulp_id       | 4e9eff35-2810-48cd-a770-067da2e249da
artifact_id   | 4e9eff35-2810-48cd-a770-067da2e249da
file          | artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
relative_path | perl-YAML1.248201812141755588652dbebx86_64snippet


NOTE: Nothing except "Complete Sync" will fix it after the changes in DB.

Comment 25 errata-xmlrpc 2022-11-16 13:34:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506

Note You need to log in before you can comment on or make changes to this bug.