Bug 1932486 - Unable to delete Red Hat repository not in manifest from Satellite
Summary: Unable to delete Red Hat repository not in manifest from Satellite
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.8.0
Hardware: Unspecified
OS: Unspecified
high
medium vote
Target Milestone: 6.10.0
Assignee: Ian Ballou
QA Contact: Cole Higgins
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-24 17:36 UTC by Jason Dickerson
Modified: 2022-05-30 13:51 UTC (History)
8 users (show)

Fixed In Version: tfm-rubygem-katello-4.1.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 14:10:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 32582 0 High Closed Unable to delete Red Hat repository due to "bad" root repo data from older Katello 2021-07-07 11:14:34 UTC
Red Hat Knowledge Base (Solution) 6301451 0 None None None 2022-05-30 13:37:34 UTC
Red Hat Product Errata RHSA-2021:4702 0 None None None 2021-11-16 14:10:21 UTC

Description Jason Dickerson 2021-02-24 17:36:11 UTC
Description of problem:

Unable to delete a Red Hat Repository


Version-Release number of selected component (if applicable):

Satellite 6.8.3


How reproducible:

Consistently


Steps to Reproduce:
1.  You have the RHEL 8 Base OS and Appstream Kicstart repositories release version "8" synced in from the release of RHEL 8 while Satellite was version 6.7.
2.  hammer repository-set disable --basearch x86_64 --id 7446 --product-id 282 --organization "Default Organization" --releasever 8

Actual results:

Could not disable repository:
  Repository not found

Expected results:

The repository is deleted.

Additional info:

Here is the pertinent configuration from my satellite 6.8.3

root@sat6 ~]# hammer repository-set available-repositories --product-id 282 --id 7446 --organization-id 1
--------------------------------------------------------------|---------|--------
NAME                                                          | RELEASE | ENABLED
--------------------------------------------------------------|---------|--------
Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart) | 8.3     | yes    
Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart) | 8.2     | no     
Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart) | 8.1     | no     
Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart) | 8.0     | no     
--------------------------------------------------------------|---------|--------
[root@sat6 ~]# hammer repository-set info --product-id 282 --id 7446 --organization-id 1
ID:                   7446
Name:                 Red Hat Enterprise Linux 8 for x86_64 - AppStream (Kickstart)
Type:                 kickstart
URL:                  /content/dist/rhel8/$releasever/x86_64/appstream/kickstart
GPG Key:              file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Label:                rhel-8-for-x86_64-appstream-kickstart
Enabled Repositories: 
 1) ID:   1486
    Name: Red Hat Enterprise Linux 8 for x86_64 - AppStream Kickstart 8.3
 2) ID:   1020
    Name: Red Hat Enterprise Linux 8 for x86_64 - AppStream Kickstart x86_64 8

[root@sat6 ~]# hammer repository-set available-repositories --product-id 282 --id 7421 --organization-id 1
-----------------------------------------------------------|---------|--------
NAME                                                       | RELEASE | ENABLED
-----------------------------------------------------------|---------|--------
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart) | 8.3     | yes    
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart) | 8.2     | no     
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart) | 8.1     | no     
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart) | 8.0     | no     
-----------------------------------------------------------|---------|--------
[root@sat6 ~]# hammer repository-set info --product-id 282 --id 7421 --organization-id 1
ID:                   7421
Name:                 Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Kickstart)
Type:                 kickstart
URL:                  /content/dist/rhel8/$releasever/x86_64/baseos/kickstart
GPG Key:              file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Label:                rhel-8-for-x86_64-baseos-kickstart
Enabled Repositories: 
 1) ID:   1018
    Name: Red Hat Enterprise Linux 8 for x86_64 - BaseOS Kickstart x86_64 8
 2) ID:   1487
    Name: Red Hat Enterprise Linux 8 for x86_64 - BaseOS Kickstart 8.3

[root@sat6 ~]# hammer repository-set disable --basearch x86_64 --id 7446 --product-id 282 --organization "Default Organization" --releasever 8
Could not disable repository:
  Repository not found
[root@sat6 ~]# hammer repository-set disable --basearch x86_64 --id 7421 --product-id 282 --organization "Default Organization" --releasever 8
Could not disable repository:
  Repository not found
[root@sat6 ~]# 

I believe this is due to the release version "8" repositories no longer being listed in the manifest.

Comment 1 David Critch 2021-03-15 18:50:38 UTC
I find myself in a similar spot. Our disk filled up and seems to have corrupted the RHEL 8.3 kickstart trees, so attempting to kickstart with 8.3 errors out about missing packages.

If I try and delete them in the UI, I get "Repository not found". 

If I try and enable in the Satellite UI, foreman stack traces since remnants of the repo still exist:
Body: {"http_request_method": "POST", "exception": null, "error_message": "Relative URL [Systems_Engineering/Library/content/dist/rhel8/8.3/x86_64/appstream/kickstart] for repository [aadf6820-7ca5-477e-a99e-bcd2b19b59ad] conflicts with existing relative URL [Systems_Engineering/Library/content/dist/rhel8/8.3/x86_64/appstream/kickstart] for repository [4915c9af-4916-4c36-9fbc-b9cc14fd22de]", "_href": "/pulp/api/v2/repositories/", "http_status": 400, "error": {"code": "PLP0000", "data": {}, "description": "Relative URL [Systems_Engineering/Library/content/dist/rhel8/8.3/x86_64/appstream/kickstart] for repository [aadf6820-7ca5-477e-a99e-bcd2b19b59ad] conflicts with existing relative URL [Systems_Engineering/Library/content/dist/rhel8/8.3/x86_64/appstream/kickstart] for repository [4915c9af-4916-4c36-9fbc-b9cc14fd22de]", "sub_errors": []}, "args": ["Relative URL [Systems_Engineering/Library/content/dist/rhel8/8.3/x86_64/appstream/kickstart] for repository [aadf6820-7ca5-477e-a99e-bcd2b19b59ad] conflicts with existing relative URL [Systems_Engineering/Library/content/dist/rhel8/8.3/x86_64/appstream/kickstart] for repository [4915c9af-4916-4c36-9fbc-b9cc14fd22de]"], "traceback": null}

If I try to delete with hammer, I get:
Could not delete the Repository:
  Red Hat repositories cannot be manipulated.

I did the repository-set command from the original comment, and see the same result so I think the issues may be related?
#  hammer repository-set disable --id 7446 --organization-id 1
Could not disable repository:
  Repository not found

Comment 2 Vincent S. Cojot 2021-04-13 17:58:48 UTC
I'm experiencing something like this on Satellite 6.8.5 system.
I cannot REMOVE repo-id 129657.
Here's the repo (note the dupe x86_64_:

[root@sat6 ~]# hammer repository list|grep 129657
129657 | Advanced Virtualization for RHEL 8 x86_64 RPMs x86_64                            | Red Hat Enterprise Linux Advanced Vir
tualization                                | yum          | https://cdn.redhat.com/content/dist/layered/rhel8/x86_64/advanced-vir
t/os       


But the actual -real- repo for that is 319096:

[root@sat6 ~]# hammer repository list|grep 'Advanced Virtualization for RHEL 8 x86_64'
319096 | Advanced Virtualization for RHEL 8 x86_64 RPMs                                   | Red Hat Enterprise Linux Advanced Vir
tualization                                | yum          | https://cdn.redhat.com/content/dist/layered/rhel8/x86_64/advanced-vir
t/os       
129657 | Advanced Virtualization for RHEL 8 x86_64 RPMs x86_64                            | Red Hat Enterprise Linux Advanced Vir
tualization                                | yum          | https://cdn.redhat.com/content/dist/layered/rhel8/x86_64/advanced-vir
t/os       

How can I get rid of that 129657 repo, please? It's been killing my Satellite refreshes.

Comment 3 Vincent S. Cojot 2021-04-13 18:07:34 UTC
[root@sat6 ~]#  hammer repository delete --id 129657
Could not delete the Repository:
  Red Hat repositories cannot be manipulated.

Comment 4 Ian Ballou 2021-05-07 18:57:56 UTC
@vcojot@redhat.com (or anyone else in this BZ having the issue), I'd like to see some more info from your Satellite's DB if possible:

1) Find the ID for the repository that you're trying to disable via hammer, the UI, or the console (in Vincent's case it's 129657)
2) Find the ID for the product that the above repository belongs to
2) Open up the foreman console (`foreman-rake console`)
3) Show me all of the repository's root info with this command: `root = ::Katello::Repository.find(<repo id from above>).root`
  -> I want to check the root's `arch` and `minor`.  These are used for matching what repository to disable, and I'm suspicious that they contain something unexpected which could cause the `Repository not found` error.
4) Show me the result of this command too: `Katello::Repository.where(root: root, environment_id: ::Katello::Product.find(<product id from above>).organization.library.id).first`

Comment 5 Jason Dickerson 2021-05-10 18:28:19 UTC
@iballou@redhat.com  I already removed the bad RHEL 8 appstream and baseos kickstarts, but just discovered my normal appstream and baseos repos are bad too.  Here is the information you requested:  

2 Bad Repositories:

Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs x86_64 8
Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs x86_64 8


[root@sat6 ~]# hammer repository list |grep "Linux 8"
1486 | Red Hat Enterprise Linux 8 for x86_64 - AppStream Kickstart 8.3                  | Red Hat Enterprise Linux for x86_64            | yum          | https://cdn.redhat.com/content/dist/rhel8/8.3/x86_64/appstream/kickstart        
1019 | Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs x86_64 8                  | Red Hat Enterprise Linux for x86_64            | yum          | https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os                 
1487 | Red Hat Enterprise Linux 8 for x86_64 - BaseOS Kickstart 8.3                     | Red Hat Enterprise Linux for x86_64            | yum          | https://cdn.redhat.com/content/dist/rhel8/8.3/x86_64/baseos/kickstart           
1017 | Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs x86_64 8                     | Red Hat Enterprise Linux for x86_64            | yum          | https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os                    
1016 | Red Hat Enterprise Linux 8 for x86_64 - Supplementary RPMs x86_64 8              | Red Hat Enterprise Linux for x86_64            | yum          | https://cdn.redhat.com/content/dist/rhel8/8/x86_64/supplementary/os             
[root@sat6 ~]# 


Bad Repository ID's:  1019 and 1017


[root@sat6 ~]# hammer product list --organization-id 1 --enabled true
----|------------------------------------------------|-------------|----------------------|--------------|------------------
ID  | NAME                                           | DESCRIPTION | ORGANIZATION         | REPOSITORIES | SYNC STATE       
----|------------------------------------------------|-------------|----------------------|--------------|------------------
143 | EPEL 6                                         |             | Default Organization | 1            | Syncing Complete.
144 | EPEL 7                                         |             | Default Organization | 1            | Syncing Complete.
310 | EPEL 8                                         |             | Default Organization | 1            | Syncing Complete.
146 | OVAL Files                                     |             | Default Organization | 1            | Syncing Complete.
145 | PuppetLabs                                     |             | Default Organization | 1            | Syncing Complete.
282 | Red Hat Enterprise Linux for x86_64            |             | Default Organization | 6            | Syncing Complete.
131 | Red Hat Enterprise Linux Server                |             | Default Organization | 12           | Syncing Complete.
39  | Red Hat Software Collections (for RHEL Server) |             | Default Organization | 2            | Syncing Complete.
----|------------------------------------------------|-------------|----------------------|--------------|------------------
[root@sat6 ~]# 


Bad Repositories belong to Product ID 282


[root@sat6 ~]# foreman-rake console
Loading production environment (Rails 6.0.3.4)
irb(main):001:0> root = ::Katello::Repository.find(1019).root
=> #<Katello::RootRepository id: 28, name: "Red Hat Enterprise Linux 8 for x86_64 - AppStream ...", created_at: "2019-05-22 20:08:50", updated_at: "2019-05-22 20:08:50", major: 8, minor: "8", gpg_key_id: nil, content_id: "7441", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_8_for_x86_64_-_AppStream_...", url: "https://cdn.redhat.com/content/dist/rhel8/8/x86_64...", unprotected: false, content_type: "yum", product_id: 282, checksum_type: nil, pulp_scratchpad_checksum_type: nil, docker_upstream_name: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignorable_content: nil, docker_tags_whitelist: nil, description: nil, ansible_collection_requirements: nil, http_proxy_policy: "global_default_http_proxy", http_proxy_id: nil, download_concurrency: nil, os_versions: nil>
irb(main):002:0> root = ::Katello::Repository.find(1017).root
=> #<Katello::RootRepository id: 26, name: "Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPM...", created_at: "2019-05-22 20:08:37", updated_at: "2019-05-22 20:08:37", major: 8, minor: "8", gpg_key_id: nil, content_id: "7416", arch: "x86_64", label: "Red_Hat_Enterprise_Linux_8_for_x86_64_-_BaseOS_RPM...", url: "https://cdn.redhat.com/content/dist/rhel8/8/x86_64...", unprotected: false, content_type: "yum", product_id: 282, checksum_type: nil, pulp_scratchpad_checksum_type: nil, docker_upstream_name: nil, mirror_on_sync: true, download_policy: "on_demand", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, ostree_upstream_sync_policy: nil, ostree_upstream_sync_depth: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignorable_content: nil, docker_tags_whitelist: nil, description: nil, ansible_collection_requirements: nil, http_proxy_policy: "global_default_http_proxy", http_proxy_id: nil, download_concurrency: nil, os_versions: nil>
irb(main):003:0> Katello::Repository.where(root: root, environment_id: ::Katello::Product.find(282).organization.library.id).first
=> #<Katello::Repository id: 1017, pulp_id: "64c208a8-9bde-4d29-8d14-d3dd7b33b52a", library_instance_id: nil, content_view_version_id: 1, relative_path: "Default_Organization/Library/content/dist/rhel8/8/...", environment_id: 1, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 26, remote_href: nil, publication_href: nil, version_href: nil>
irb(main):004:0> quit
[root@sat6 ~]#

Comment 6 Vincent S. Cojot 2021-05-11 12:36:52 UTC
@iballou@redhat.com : I am sorry, I ended up clearing the issue on my Satellite because I needed to recover functionality. My satellite no longer has that bogus repo.

Comment 7 Vincent S. Cojot 2021-05-11 12:39:46 UTC
Here is how I resolved the issue:

# foreman-rake console        ----> Let it load.
irb(main:001:0> User.current = User.find_by_login('admin')
irb(main:002:0> ForemanTasks.sync_task(::Actions::Katello::Repository::Destroy, Katello::Repository.find(129657))

# foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null

Comment 8 Ian Ballou 2021-05-11 13:27:02 UTC
Thanks Vincent, no worries. I have a DB dump from Jason now that should give me enough to solve the bug.

Comment 9 Ian Ballou 2021-05-11 21:07:38 UTC
Created redmine issue https://projects.theforeman.org/issues/32582 from this bug

Comment 10 Bryan Kearney 2021-06-02 16:03:56 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/32582 has been resolved.

Comment 16 errata-xmlrpc 2021-11-16 14:10:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702


Note You need to log in before you can comment on or make changes to this bug.