Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2095314

Summary: satellite-change-hostname too destructive when it fails due to unability to forward resolve the new hostname
Product: Red Hat Satellite Reporter: Lukáš Hellebrandt <lhellebr>
Component: satellite-change-hostnameAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.11.0CC: ehelms
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-01 21:00:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukáš Hellebrandt 2022-06-09 14:05:15 UTC
Description of problem:
When running spacewalk-change-hostname without the new FQDN actually being forward resolvable by the DNS (i.e. attempting to only run locally), the action fails. I don't consider the fact it failed a bug, but the way it fails is quite destructive.

After the failure, the Satellite doesn't run, I can't connect to WebUI.
After `foreman-maintain service restart`, there is an issue with Candlepin, WebUI only works to throw this error: `Oops, we're sorry but something went wrong A backend service [ Candlepin ] is unreachable`
Attempting to revert the changes to get a working satellite again:
```
# hammer ping
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-server-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://fish.example.com.

Warning: An error occured while loading module hammer_cli_foreman.
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/pki/katello/certs/katello-server-ca.crt

Make sure the location contains an unexpired and valid CA certificate for https://fish.example.com.

[...]

Warning: An error occured while loading module hammer_cli_katello.
Error: No such sub-command 'ping'.

See: 'hammer --help'.
```
Trying to workaround by manually changing hostname doesn't help either:
# hostnamectl set-hostname dhcp-3-121.vms.sat.rdu2.redhat.com
# foreman-maintain service restart
=> restarts successfully, but in WebUI: `Oops, we're sorry but something went wrong A backend service [ Candlepin ] is unreachable`

Because I ran a tool without DNS being able to resolve my new FQDN, I ended up with a bricked Satellite. Perhaps it can be fixed but this is way too destructive for such a simple mishap.

The log of satellite-change-hostname was:
```
# satellite-change-hostname -uadmin -p<PASSWORD> fish.example.com
[...]
stopping services
removing old cert rpms
No Match for argument: dhcp-3-121.vms.sat.rdu2.redhat.com-apache*
No Match for argument: dhcp-3-121.vms.sat.rdu2.redhat.com-foreman-client*
No Match for argument: dhcp-3-121.vms.sat.rdu2.redhat.com-foreman-proxy*
[...]
deleting old certs
backed up /var/www/html/pub to /var/www/html/pub/dhcp-3-121.vms.sat.rdu2.redhat.com-20220609093929.backup
updating hostname in /etc/hosts
updating hostname in foreman installer scenarios
updating hostname in hammer configuration
backing up last_scenario.yaml
removing last_scenario.yaml
re-running the installer
satellite-installer --scenario satellite -v --disable-system-checks --certs-regenerate=true --foreman-proxy-register-in-foreman true
Output of 'facter fqdn' is different from 'hostname -f'

Make sure above command gives the same output. If needed, change the hostname permanently via the
'hostname' or 'hostnamectl set-hostname' command
and editing the appropriate configuration file.
(e.g. on Red Hat systems /etc/sysconfig/network,
on Debian based systems /etc/hostname).

If 'hostname -f' still returns an unexpected result, check /etc/hosts and put
the hostname entry in the correct order, for example:

  1.2.3.4 hostname.example.com hostname

The fully qualified hostname must be the first entry on the line
[...]
  Something went wrong with the Satellite installer.
  Please check the above output and the corresponding logs.
  
  Once the issue is resolved you may complete the hostname change[...quite complicated steps...]
```

Version-Release number of selected component (if applicable):
6.11 snap 23, probably not a regression

How reproducible:
Deterministic

Steps to Reproduce:
# satellite-change-hostname nonsense-hostname.example.com -uadmin -p<PASSWORD>


Actual results:
Destructive failure

Expected results:
Graceful failure, with a simplish way to revert the changes and get a working Satellite with the original hostname.
I would also like there to be a --local-only switch that would ignore the fact that I am using a FQDN that nobody will be able to reach me with.

Additional info:
See also bug 1861831 which also shows inability to re-run spacewalk-change-hostname upon failure, the bug has been fixed but it's effectively still not possible.
Setting severity medium because although this bricks a Satellite, it is due to previous mistake by the user, in a not-so-common workflow and is probably recoverable.

Comment 1 Brad Buckingham 2023-07-21 21:06:39 UTC
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team.  Thank you.

Comment 2 Brad Buckingham 2023-09-01 21:00:04 UTC
Thank you for your interest in Red Hat Satellite. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to contact your Red Hat Account Team. Thank you.