Bug 1720369
| Summary: | Running normal maintenance results in broken symlinks for repository metadata. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Jason Dickerson <jdickers> | ||||
| Component: | Repositories | Assignee: | Partha Aji <paji> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Cole Higgins <chiggins> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 6.5.0 | CC: | ahumbe, alexandre.chanu, dalley, dkliban, gpayelka, ktordeur, mvanderw, paji, pdwyer, spetrosi, val.baranov | ||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||
| Target Release: | Unused | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Known Issue | |||||
| Doc Text: |
Entering the following command can lead to broken symlinks for repository metadata. Do not enter this command until BZ#1720369 is resolved.
+
----
# foreman-rake katello:delete_orphaned_content RAILS_ENV=production
----
+
If you have broken symlinks, regenerate the yum repository metadata.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-09-14 19:49:13 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Some observations on the broken symlink list. 398 rpms that are/were part of EPEL 6. The broken symlinks were part of a Composite Content View for RHEL 6.5 EUS, which has not been published in quite some time. The EPEL repos are set to mirror, so as rpms are removed upstream, and the repo is synced, the rpms are removed from Satellite. The customer would expect this to not result in broken symlinks. ie if the rpms are in use, do not remove them from pulp, even though they are removed upstream. 73 Delta Rpms that are part of SLES repositories. The customer has put a filter in place on the repositories to not sync in drpm's. This has not always been in place, but has been in place for several months. 110 repodata products.xml or products.xml.gz. SLES repodata 78 patterns.xml. SLES repodata 78 license.tar.gz. SLES repodata 51 appdata.xml.gz. SLES repodata productid.gz from Kickstarts: 6.10, 7.5, and 7.4 repomd.xml from the HighAvailability, LoadBalancer, ResilientStorage, and ScalableFileSystem repos in the 6.10 kickstart We have not re-published all CV's and CCV's since the code patch was put in place. Some of the CCV's and CV's are on quarterly schedules and some are ad-hoc. We will have to continue to monitor until all CCV's and CV's are re-published/promoted. Some questions remain: Would the 398 epel rpms missing be due to the mirroring of the upstream, and pulp removed the rpm, even though they were part of a CV or CCV? Will the 73 Delta rpms be an issue until all CV and CCV's using those repositories are republished? Do we expect to continue encountering broken symlinks with the kickstart repositories? https://bugzilla.redhat.com/show_bug.cgi?id=1691105 addresses yum metadata files (i.e. files under repodata/ directory). It wont affect rpms. if 398 rpms were missing that is a different issue. Do you see any patterns in the rpms that were missing. Like do they belong to the same cvv etc ? 398 rpms that are/were part of EPEL 6. The broken symlinks were part of a Composite Content View for RHEL 6.5 EUS, which has not been published in quite some time. The EPEL repos are set to mirror, so as rpms are removed upstream, and the repo is synced, the rpms are removed from Satellite. ------------------------------ Since this was noted above, we are now seeing broken symlinks for rpm's in EPEL 7. Our best guess is that as we are mirroring the upstream and the upstream removes rpms, the syncs are removing the rpms from pulp, creating a situation where existing CV's and CCV's which link to these rpms, now have broken symlinks. Also, we have noticed a number of broken symlinks belong to pulp repositores for CV and CCV versions, that no longer exist in Satellite/katello. It seems that when these CV and CCV versions are deleted, the pulp repositories are being left behind indefinitely. Is the Epel 6 Misc repo on lazy sync ? That could possibly explain discrepancies. It is currently set to immediate. However, the first time it sync'ed it may have been set to on-demand/lazy sync, as on-demand/lazy was the default at the time. Linked 3 bugs in reference to this 1) 1804977 - drpms not getting copied over cv publish - This should fix the broken drpm symlink 2) 1691105 - Content view version delete results in broken sym links - This should handle everything in the repodata/* directories 3) 1803988 - Changing on-demand to immediate not working with broken symlinks - Applicable only if repo first synced with lazy and then changed to immediate. Possibly in the capsule. Satellite 6.10+ doesn't use symlinks for metadata, so I'm closing this. Bug isn't possible anymore. |
Created attachment 1580396 [details] list of broken symlinks in json format Description of problem: Whenever we run: foreman-rake katello:delete_orphaned_content RAILS_ENV=production We start to notice broken symlinks for repository metadata. The only workaround is to identify the repositories and regenerate the yum repository metadata for each one. I will be attaching the list of broken symlinks. Where are you experiencing the behavior? What environment? Production When does the behavior occur? Frequently? Repeatedly? At certain times? frequently What information can you provide around timeframes and the business impact? This causes clients to encounter yum errors, preventing the installation/upgrading of packages. Version-Release number of selected component (if applicable): How reproducible: Consistently with non-RHEL OS's Steps to Reproduce: 1.sync new content 2.publish and promote CV's 3.run foreman-rake katello:delete_orphaned_content RAILS_ENV=production Actual results: many broken symlinks Expected results: no broken symlinks Additional info: