Bug 2131415

Summary: After changing Satellite hostname Capsules still have packages referring to old hostname
Product: Red Hat Satellite Reporter: Paul Dudley <pdudley>
Component: satellite-change-hostnameAssignee: satellite6-bugs <satellite6-bugs>
Status: NEW --- QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.11.2CC: ahumbe, ehelms
Target Milestone: UnspecifiedKeywords: Triaged, WorkAround
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paul Dudley 2022-09-30 23:39:22 UTC
On Satellite-capsule 6.11.2 rhel 8, receive this error when a content host registered to the Capsule attempts to install a package:
~~~
Sep 27 12:27:34 oldname pulpcore-content[523678]: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'oldname.example.com'. (_ssl.c:1131)
~~~

Looking into this, we find that all remotes on the Capsules are correct - they reference the correct new name:
~~~
From Sat to Cap:
- # curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://capsule.example.com/pulp/api/v3/remotes/rpm/rpm/ | python -m json.tool | grep \"url\"\:
            "url": "https://newname.example.com/pulp/content/ghost/prod/rhel9/content/dist/rhel9/9/x86_64/appstream/os/"
            "url": "https://newname.example.com/pulp/content/ghost/prod/rhel9/content/dist/rhel9/9/x86_64/baseos/os/"
            "url": "https://newname.example.com/pulp/content/ghost/prod/rhel9/content/dist/layered/rhel9/x86_64/sat-client/6/os/"
            "url": "https://newname.example.com/pulp/content/ghost/prod/rhel8/content/dist/rhel8/8/x86_64/appstream/os/"
            "url": "https://newname.example.com/pulp/content/ghost/prod/rhel8/content/dist/rhel8/8/x86_64/baseos/os/"
            "url": "https://newname.example.com/pulp/content/ghost/prod/rhel8/content/dist/layered/rhel8/x86_64/sat-tools/6.10/os/"
            "url": "https://newname.example.com/pulp/content/ghost/prod/SmallCV/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.6/os/"
...

- # curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://capsule.example.com/pulp/api/v3/remotes/rpm/rpm/ | python -m json.tool | grep \"url\"\: | grep oldname | wc
      0       0       0 <-- no results for the old Satellite name
...

- # curl --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://capsule.example.com/pulp/api/v3/remotes/rpm/rpm/ | python -m json.tool | grep \"url\"\: | grep newname | wc
      7      14     930 <-- all remotes have new Satellite name
~~~

However, if we look at the remoteartifacts, plenty of them still reference the old Satellite name:
~~~
On Capsule:
- # su - postgres -c "psql pulpcore -c \"select url from core_remoteartifact;\"" | grep oldname.example.com | wc
some number (not 0)

- # su - postgres -c "psql pulpcore -c \"select url from core_remoteartifact;\"" | grep newname.example.com | wc
some number (also not 0)
~~~

It looks like after changing the Satellite hostame, and subsequent needed changes on the Capsules, like successful syncs, the url on remotes themselves are changed as we expect, but urls on remoteartifacts are not updated. This means that any content previously synced on the Capsule(s) before the change will look for the wrong Satellite name to download the rpm from.

Expectation here would be that after changing the Satellite hostname, all necessary changes are made on Capsule side to change all content to reference the new Satellite name.

To reproduce:
- Install Satellite and Capsule
- Add content to Satellite, sync to Capsule
- Change Satellite hostname
- Sync Capsule to find that remotes are all updated correctly
- Add more content to Satellite, sync that content to both Sat and Cap
- Check remote artifacts to find that there are references to both the old and new Satellite name

Comment 1 Paul Dudley 2022-09-30 23:42:35 UTC
To update remote artifacts You can run this (assuming rhel8):
~~~
On Capsule:
- # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager dbshell
pulpcore=> BEGIN;
pulpcore=> update "core_remoteartifact"
pulpcore=> set url = REPLACE(url, 'https://oldname.example.com','https://newname.example.com');
- verified old name no longer found - if problems, ROLLBACK; -
COMMIT;
~~~