*** Bug 2158896 has been marked as a duplicate of this bug. ***

I was finally able to reproduce this bug.

It only happens:
- for backups that have either Puppet or Katello-Agent/Qpidd features enabled
- when restoring directly with foreman-maintain (and not satellite-clone) (yes, this is absolutely supported, just limits the impact)
- when restoring to a system that does not yet have the same features enabled (this is *technically* unsupported, as we document in [1] that the system to restore to needs to have "the same configuration", but do not elaborate exactly which bits need to be "same")

The issue is when the puppetserver or qpid-cpp-server packages are not installed while we unpack the backup, the files that should be owned by `puppet` or `qpidd` are extracted with their numeric ownership.
Would the packages (and thus the users) be already present, tar would be able to look up the correct UID/GID combination (as the tarball *contains* the names!) and the restore would work.

satellite-clone avoids this issue, as it checks whether the backup has puppet/qpidd and pre-installs the packages [2].

a viable workaround for users who do not wish to use satellite-clone is to install puppetserver/qpid-cpp-server on the system before running the restore, or follow the documentation to enable those features on the target system before doing the restore (but really, installing the packages is enough).

[1] https://access.redhat.com/documentation/en-us/red_hat_satellite/6.11/html/administering_red_hat_satellite/restoring_server_or_smart_proxy_from_a_backup_admin
[2] https://github.com/RedHatSatellite/satellite-clone/commit/8b70ae2b66b7f1cb125cf5868b3b4397618a5990

Created redmine issue https://projects.theforeman.org/issues/36578 from this bug

I've written how to reproduce this without el7/el8 on a 6.14 in https://github.com/theforeman/foreman_maintain/pull/744#issuecomment-1630638289

VERIFIED.

@Satellite 6.14.0 Snap 13
rubygem-foreman_maintain-1.3.5-1.el8sat.noarch

by the following manual reproducer:

1) Enable katello-agent feature 
# satellite-installer --foreman-proxy-content-enable-katello-agent true

2) Create a backup
# satellite-maintain backup offline -y /var/backup

3) On another machine with the same hostname install satellite with the defaults
(=katello-agent is disabled and not installed)

4) Restore from the backup made on 1st machine
# satellite-maintain restore -y /var/backup/satellite-backup-2023-08-29-08-04-05

REPRO:

--------------------------------------------------------------------------------
Restore configs from backup: 
- Restoring configs                                                   [OK]      
--------------------------------------------------------------------------------
Run installer reset: 
| Installer reset                                                     [FAIL]    
Failed executing yes | satellite-installer -v --reset-data , exit status 6:
...
2023-08-29 12:11:39 [ERROR ] [configure] Systemd start for qpidd failed!
2023-08-29 12:11:39 [ERROR ] [configure] journalctl log for qpidd:
2023-08-29 12:11:39 [ERROR ] [configure] -- Logs begin at Tue 2023-08-29 11:19:02 EDT, end at Tue 2023-08-29 12:11:39 EDT. --
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:10:09 satellite.example.com systemd[1]: Starting An AMQP message broker daemon....
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:10:09 satellite.example.com qpidd[39796]: 2023-08-29 12:10:09 [Broker] critical Unexpected error: Cannot open lock file /var/lib/qpidd/lock: Permission denied
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:10:09 satellite.example.com qpidd[39796]: 2023-08-29 12:10:09 [Broker] critical Unexpected error: Cannot open lock file /var/lib/qpidd/lock: Permission denied
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:10:09 satellite.example.com systemd[1]: qpidd.service: Main process exited, code=exited, status=1/FAILURE
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:11:39 satellite.example.com systemd[1]: qpidd.service: Start-post operation timed out. Stopping.
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:11:39 satellite.example.com systemd[1]: qpidd.service: Failed with result 'exit-code'.
2023-08-29 12:11:39 [ERROR ] [configure] Aug 29 12:11:39 satellite.example.com systemd[1]: Failed to start An AMQP message broker daemon..
2023-08-29 12:11:39 [ERROR ] [configure] /Stage[main]/Qpid::Service/Service[qpidd]/ensure: change from 'stopped' to 'running' failed: Systemd start for qpidd failed!

vs.

FIX:

--------------------------------------------------------------------------------
Ensure required packages are installed before restore: 
/ Installing required packages                                        [OK]      
--------------------------------------------------------------------------------
Restore configs from backup: 
/ Restoring configs                                                   [OK]      
--------------------------------------------------------------------------------
Run installer reset: 
| Installer reset                                                     [OK]      
--------------------------------------------------------------------------------

>>> restore now ensures required users/groups exist prior restoring configs (in order to map users/groups in archive vs. in system)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818