Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1720369 - Running normal maintenance results in broken symlinks for repository metadata.
Summary: Running normal maintenance results in broken symlinks for repository metadata.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.5.0
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: Unspecified
Assignee: Partha Aji
QA Contact: Cole Higgins
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-13 20:18 UTC by Jason Dickerson
Modified: 2024-12-20 18:51 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Entering the following command can lead to broken symlinks for repository metadata. Do not enter this command until BZ#1720369 is resolved. + ---- # foreman-rake katello:delete_orphaned_content RAILS_ENV=production ---- + If you have broken symlinks, regenerate the yum repository metadata.
Clone Of:
Environment:
Last Closed: 2022-09-14 19:49:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
list of broken symlinks in json format (24.13 KB, application/gzip)
2019-06-13 20:18 UTC, Jason Dickerson 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1691105 0 unspecified CLOSED Content view version delete results in broken sym links 2023-10-06 18:11:43 UTC
Red Hat Bugzilla 1803988 0 unspecified CLOSED Changing on-demand to immediate not working with broken symlinks 2024-12-20 19:00:00 UTC
Red Hat Bugzilla 1804977 0 unspecified CLOSED drpms not getting copied over cv publish 2021-02-22 00:41:40 UTC

Description Jason Dickerson 2019-06-13 20:18:46 UTC 
Created attachment 1580396 [details]
list of broken symlinks in json format

Description of problem:

Whenever we run:

foreman-rake katello:delete_orphaned_content RAILS_ENV=production

We start to notice broken symlinks for repository metadata.  

The only workaround is to identify the repositories and regenerate the yum repository metadata for each one.  

I will be attaching the list of broken symlinks.

Where are you experiencing the behavior?  What environment?

Production

When does the behavior occur? Frequently?  Repeatedly?   At certain times?

frequently

What information can you provide around timeframes and the business impact?

This causes clients to encounter yum errors, preventing the installation/upgrading of packages.

Version-Release number of selected component (if applicable):


How reproducible:
Consistently with non-RHEL OS's

Steps to Reproduce:
1.sync new content
2.publish and promote CV's
3.run foreman-rake katello:delete_orphaned_content RAILS_ENV=production


Actual results:
many broken symlinks

Expected results:
no broken symlinks

Additional info:
Comment 4 Jason Dickerson 2019-06-14 15:05:56 UTC 
Some observations on the broken symlink list.  

398 rpms that are/were part of EPEL 6.  The broken symlinks were part of a Composite Content View for RHEL 6.5 EUS, which has not been published in quite some time.  The EPEL repos are set to mirror, so as rpms are removed upstream, and the repo is synced, the rpms are removed from Satellite.  

The customer would expect this to not result in broken symlinks.  ie if the rpms are in use, do not remove them from pulp, even though they are removed upstream.  

73 Delta Rpms that are part of SLES repositories.  The customer has put a filter in place on the repositories to not sync in drpm's.  This has not always been in place, but has been in place for several months.  

110 repodata products.xml or products.xml.gz.  SLES repodata

78 patterns.xml.  SLES repodata

78 license.tar.gz.  SLES repodata

51 appdata.xml.gz.  SLES repodata

productid.gz from Kickstarts: 6.10, 7.5, and 7.4

repomd.xml from the HighAvailability, LoadBalancer, ResilientStorage, and ScalableFileSystem repos in the 6.10 kickstart
Comment 5 Jason Dickerson 2019-06-14 15:58:22 UTC 
We have not re-published all CV's and CCV's since the code patch was put in place.  Some of the CCV's and CV's are on quarterly schedules and some are ad-hoc.  We will have to continue to monitor until all CCV's and CV's are re-published/promoted.  

Some questions remain:  

Would the 398 epel rpms missing be due to the mirroring of the upstream, and pulp removed the rpm, even though they were part of a CV or CCV?  

Will the 73 Delta rpms be an issue until all CV and CCV's using those repositories are republished?  

Do we expect to continue encountering broken symlinks with the kickstart repositories?
Comment 6 Partha Aji 2019-06-14 17:21:45 UTC 
https://bugzilla.redhat.com/show_bug.cgi?id=1691105 addresses yum metadata files (i.e. files under repodata/ directory). It wont affect rpms. if 398 rpms were missing that is a different issue. 
Do you see any patterns in the rpms that were missing. Like do they belong to the same cvv etc ?
Comment 7 Jason Dickerson 2019-06-19 14:14:11 UTC 
  398 rpms that are/were part of EPEL 6.  The broken symlinks were part of a Composite Content View for RHEL 6.5 EUS, which has not been published in quite some time.  The EPEL repos are set to mirror, so as rpms are removed upstream, and the repo is synced, the rpms are removed from Satellite.  

------------------------------

Since this was noted above, we are now seeing broken symlinks for rpm's in EPEL 7.

Our best guess is that as we are mirroring the upstream and the upstream removes rpms, the syncs are removing the rpms from pulp, creating a situation where existing CV's and CCV's which link to these rpms, now have broken symlinks.  

Also, we have noticed a number of broken symlinks belong to pulp repositores for CV and CCV versions, that no longer exist in Satellite/katello.  It seems that when these CV and CCV versions are deleted, the pulp repositories are being left behind indefinitely.
Comment 9 Partha Aji 2019-06-21 15:27:28 UTC 
Is the Epel 6 Misc repo  on lazy sync ? That could possibly explain discrepancies.
Comment 10 Jason Dickerson 2019-07-02 16:51:40 UTC 
It is currently set to immediate.  However, the first time it sync'ed it may have been set to on-demand/lazy sync, as on-demand/lazy was the default at the time.
Comment 14 Partha Aji 2020-02-21 20:20:46 UTC 
Linked 3 bugs in reference to this

1) 1804977 - drpms not getting copied over cv publish  - This should fix the broken drpm symlink
2) 1691105 - Content view version delete results in broken sym links -  This should handle everything in the repodata/* directories
3) 1803988 -  Changing on-demand to immediate not working with broken symlinks - Applicable only if repo first synced with lazy and then changed to immediate. Possibly in the capsule.
Comment 21 Daniel Alley 2022-09-14 19:49:13 UTC 
Satellite 6.10+ doesn't use symlinks for metadata, so I'm closing this.  Bug isn't possible anymore.
Note You need to log in before you can comment on or make changes to this bug.