Bug 2107701

Summary: [Pulp 3] If a modulemd metadata artifact is missing from the filesystem but has an artifact_id associated with it in database, "Verify Content Checksum" cannot fix this problem
Product: Red Hat Satellite Reporter: Sayan Das <saydas>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Vladimír Sedmík <vsedmik>
Severity: medium Docs Contact:
Priority: high    
Version: 6.11.0CC: dalley, dkliban, ggainey, myarboro, rchan
Target Milestone: 6.12.0Keywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulp_rpm 3.18.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-16 13:34:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sayan Das 2022-07-15 16:59:07 UTC 
Description of problem:

With a combination of multiple failed and successful sync attempts on a repo like "Appstream 8", It may happen that, One or more artifacts related to modulemd metdata is physically missing from the filesystem but if we check the pulpcore database, then the same artifact is having an artifact_id associated with it which means pulp thinks the file is alredy created\downloaded on the filesystem.

This can happen for multiple modulemd files and an attempt of exporting the repository will fail in these circumstances:

# hammer content-export complete repository --organization="XXXX" --id='791'
[...............................................................................................................................................................................................] [100%]
Error: [Errno 2] No such file or directory: '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1'
undefined method `first' for nil:NilClass

The repair feature of pulp, Integrated with "Validate content checksum" option of Katello, shows that it was able to find and fix a missing file from the concerned "Appstream 8" repo but actually , that modulemd artifact is still missing from the filesystem.


Version-Release number of selected component (if applicable):

Satellite 6.10 and 6.11
Pulp 3


How reproducible:

Always


Steps to Reproduce:
1. Install Satellite 6.11 on RHEL 8 | or 6.10 on RHEl 7
2. Enable and sync the "Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8" repo.
3. Check for the availability of this file in filesystem i.e. /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
4. Check for the presence of this artifact artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1 in pulpcore db
5. Remove that file from filesystem.
6. "Complete Sync" the repo and check if the file came back.
7. Run a "Verify content checksum" against teh same repo and check the results.

Actual results:

On FS:

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  File: /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  Size: 2480      	Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 163610175   Links: 1
Access: (0644/-rw-r--r--)  Uid: (  988/    pulp)   Gid: (  985/    pulp)
Context: system_u:object_r:pulpcore_var_lib_t:s0
Access: 2022-07-15 02:22:18.547076698 +0530
Modify: 2022-06-08 06:32:07.524593523 +0530
Change: 2022-06-08 06:36:17.184745061 +0530
 Birth: 2022-06-08 06:32:07.524593523 +0530
 
# head /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
---
document: modulemd
version: 2
data:
  name: perl-YAML
  stream: "1.24"
  version: 820181214175558
  context: 8652dbeb
  arch: x86_64
  summary: Perl parser for YAML


in DB:

# echo "select ca.pulp_id,cca.artifact_id,ca.file,cca.relative_path from core_artifact ca LEFT JOIN core_contentartifact cca on cca.artifact_id = ca.pulp_id where ca.file = 'artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1';"   | su - postgres -c "psql -x pulpcore"
-[ RECORD 1 ]-+---------------------------------------------------------------------------
pulp_id       | 17e28edc-0195-4115-922c-863f40e7ef62
artifact_id   | 17e28edc-0195-4115-922c-863f40e7ef62
file          | artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
relative_path | perl-YAML1.248201812141755588652dbebx86_64snippet

# echo "select pulp_id,pulp_created,pulp_last_updated,file,sha256 from core_artifact where file = 'artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1';"    | su - postgres -c "psql -x pulpcore"
-[ RECORD 1 ]-----+---------------------------------------------------------------------------
pulp_id           | 17e28edc-0195-4115-922c-863f40e7ef62
pulp_created      | 2022-06-08 06:36:17.185196+05:30
pulp_last_updated | 2022-06-08 06:36:17.185208+05:30
file              | artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
sha256            | a38b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1


# echo "select * from core_contentartifact where artifact_id = '17e28edc-0195-4115-922c-863f40e7ef62';"    | su - postgres -c "psql -x pulpcore" 
-[ RECORD 1 ]-----+--------------------------------------------------
pulp_id           | 3c8a3f07-703e-4c82-835f-a3292dc83bed
pulp_created      | 2022-02-19 16:29:04.054172+05:30
pulp_last_updated | 2022-02-19 16:29:04.054176+05:30
relative_path     | perl-YAML1.248201812141755588652dbebx86_64snippet
artifact_id       | 17e28edc-0195-4115-922c-863f40e7ef62
content_id        | 05c92fad-816c-425a-8dc5-a1e40c947c87

Delete:

# rm /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
rm: remove regular file '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1'? y

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
stat: cannot statx '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1': No such file or directory

Simple Sync: No luck

Verify Content Sync: Even though it shows the missing unit has been fixed, still no luck. The file is still absent.

Products -> Red Hat Enterprise Linux for x86_64 -> Repositories -> Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8 -> Select Action -> Verify Content Checksum


Jul 15 21:01:21 satellite611 pulpcore-api[124567]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]:  - - [15/Jul/2022:15:31:21 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/3d8caef4-e94a-4092-934d-c79937a7adf9/versions/40/repair/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.17.4/ruby"
Jul 15 21:01:21 satellite611 pulpcore-api[124567]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]:  - - [15/Jul/2022:15:31:21 +0000] "GET /pulp/api/v3/tasks/6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5/ HTTP/1.1" 200 572 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 15 21:01:21 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5
..
Jul 15 21:01:21 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.app.tasks.repository:INFO: Repairing version 40 of repository 'Red_Hat_Enterprise_Linux_8_for_x86_64_-_AppStream_RPMs_8-176782'
..
Jul 15 21:01:22 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.app.tasks.repository:WARNING: Missing file for <Artifact: pk=17e28edc-0195-4115-922c-863f40e7ef62>
..
Jul 15 21:01:23 satellite611 pulpcore-api[161362]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]:  - - [15/Jul/2022:15:31:23 +0000] "GET /pulp/api/v3/tasks/6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5/ HTTP/1.1" 200 946 "-" "OpenAPI-Generator/3.16.7/ruby"
Jul 15 21:01:23 satellite611 pulpcore-worker-1[190116]: pulp [a4411352-67eb-45b6-a238-a1b17ad1edb5]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5



# curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://`hostname -f`/pulp/api/v3/tasks/6bb7ecb2-8a83-4acd-9cbc-28ab9502c7a5/ | jq .progress_reports
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   979  100   979    0     0  16316      0 --:--:-- --:--:-- --:--:-- 16593
[
  {
    "message": "Identify missing units",
    "code": "repair.missing",
    "state": "completed",
    "total": null,
    "done": 1,
    "suffix": null
  },
  {
    "message": "Identify corrupted units",
    "code": "repair.corrupted",
    "state": "completed",
    "total": null,
    "done": 0,
    "suffix": null
  },
  {
    "message": "Repair corrupted units",
    "code": "repair.repaired",
    "state": "completed",
    "total": null,
    "done": 0,
    "suffix": null
  }
]

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
stat: cannot statx '/var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1': No such file or directory


Republished Repository metadata: But that also did not helped.

Complete Sync done: That also did not helped. 


Expected results:

If modulemd metadata files are supposed to be downloaded from remote only, then repari action via "Verify Content Checksum" should work.

If modulemd metadata files are created by pulp similar to repository metadata files, then a "Republish" action on the repo via API or "Complete Sync" should be able to recreate the file.


Additional info:

The only way i can fix it is :

** nullify the artifact_id from pulpcore db core_contentartifact table and delete the record of that artifact from core_artifact

# su - postgres -c "psql pulpcore"
psql (12.11)
Type "help" for help.

pulpcore=# UPDATE core_contentartifact SET artifact_id = NULL WHERE relative_path = 'perl-YAML1.248201812141755588652dbebx86_64snippet'
pulpcore-# ;
UPDATE 1
pulpcore=# delete from  core_artifact where pulp_id = '17e28edc-0195-4115-922c-863f40e7ef62';
DELETE 1
pulpcore=# exit


** Then "Complete Sync" the repo. 

# stat /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  File: /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
  Size: 2480      	Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 306281556   Links: 1
Access: (0644/-rw-r--r--)  Uid: (  988/    pulp)   Gid: (  985/    pulp)
Context: system_u:object_r:pulpcore_var_lib_t:s0
Access: 2022-07-15 21:07:12.437638569 +0530
Modify: 2022-07-15 21:07:12.436638480 +0530
Change: 2022-07-15 21:10:36.189788367 +0530
 Birth: 2022-07-15 21:07:12.436638480 +0530


# sha256sum /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
a38b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1  /var/lib/pulp/media/artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1


# echo "select ca.pulp_id,cca.artifact_id,ca.file,cca.relative_path from core_artifact ca LEFT JOIN core_contentartifact cca on cca.artifact_id = ca.pulp_id where ca.file = 'artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1';"   | su - postgres -c "psql -x pulpcore"
-[ RECORD 1 ]-+---------------------------------------------------------------------------
pulp_id       | 4e9eff35-2810-48cd-a770-067da2e249da
artifact_id   | 4e9eff35-2810-48cd-a770-067da2e249da
file          | artifact/a3/8b3909c854fa4b0d5ec79da8e9df389475b3f2e8ace9ee79f6f99ab1f31ec1
relative_path | perl-YAML1.248201812141755588652dbebx86_64snippet


NOTE: Nothing except "Complete Sync" will fix it after the changes in DB.
Comment 25 errata-xmlrpc 2022-11-16 13:34:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506