Bug 1896038

Summary: Satellite upgrade from 6.7 to 6.8 fails
Product: Red Hat Satellite Reporter: Afeef Ghannam <afeef.ghannam>
Component: InstallationAssignee: Ewoud Kohl van Wijngaarden <ekohlvan>
Status: CLOSED ERRATA QA Contact: Devendra Singh <desingh>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: ahumbe, dhjoshi, ehelms, ekohlvan, janarula, jbhatia, kai.krieger-juhnke, ktordeur, ldelouw, saydas, sfroemer, vijsingh, zhunting
Target Milestone: 6.9.0Keywords: PrioBumpGSS, Triaged, Upgrades
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-kafo-6.1.2-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1919413 (view as bug list) Environment:
Last Closed: 2021-04-21 13:18:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Afeef Ghannam 2020-11-09 16:21:56 UTC 
Description of problem:

I have checked the upgrade from 6.7 to 6.8 everything was ok.

satellite-maintain upgrade check --target-version 6.8

Running preparation steps required to run the next scenarios
================================================================================
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled:
/ Checking repositories enabled on the systemUnable to upload Enabled Repositories Report
| Checking repositories enabled on the system                         [OK]
--------------------------------------------------------------------------------


Running Checks before upgrading to Satellite 6.8
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check to make sure root(/) partition has enough space:                [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:                                 [OK]
--------------------------------------------------------------------------------
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for old tasks in paused/stopped state:                          [OK]
--------------------------------------------------------------------------------
Check for pending tasks which are safe to delete:                     [OK]
--------------------------------------------------------------------------------
Check for tasks in planning state:                                    [OK]
--------------------------------------------------------------------------------
Check to verify if any hotfix installed on system:
\ Checking for presence of hotfix(es). It may take some time to verify.
                                                                      [OK]
--------------------------------------------------------------------------------
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled:
| Checking repositories enabled on the systemUnable to upload Enabled Repositories Report
\ Checking repositories enabled on the system                         [OK]
--------------------------------------------------------------------------------
Check if TMOUT environment variable is set:                           [OK]
--------------------------------------------------------------------------------
Check if any upstream repositories are enabled on system:
\ Checking for presence of upstream repositories                      [OK]
--------------------------------------------------------------------------------
Check if yum exclude list is configured:                              [OK]
--------------------------------------------------------------------------------
Check for roles that have filters with multiple resources attached:   [OK]
--------------------------------------------------------------------------------
Check for duplicate permissions from database:                        [OK]
--------------------------------------------------------------------------------
Check whether reports have correct associations:                      [OK]
--------------------------------------------------------------------------------
Check if checkpoint_segments configuration exists on the system:      [OK]
--------------------------------------------------------------------------------
Validate availability of repositories:
/ Validating availability of repositories for 6.8                     [OK]
--------------------------------------------------------------------------------

Then I started the upgrade satellite-maintain upgrade run --target-version 6.8 but the upgrade failed with

Procedures::Installer::Upgrade: Could not get default values, check log file at /var/log/foreman-installer/satellite.log for more information
                                      [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 25
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.8] failed.

The following steps ended up in failing state:

  [installer-upgrade]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="installer-upgrade"


In satellite.log:

 | RUBYLIB=/opt/theforeman/tfm/root/usr/share/gems/gems/kafo-4.1.0/lib/kafo/../..//modules: /usr/local/bin/puppet apply --config=/tmp/kafo_installation20201109-25213-1dh0xwi/puppet.conf
[ERROR 2020-11-09T16:46:45 main] Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring ovirt-engine-sdk-4.2.3 because its extensions are not built. Try: gem pristine ovirt-engine-sdk --version 4.2.3
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring puma-4.3.3 because its extensions are not built. Try: gem pristine puma --version 4.3.3
Ignoring qpid_messaging-1.36.0 because its extensions are not built. Try: gem pristine qpid_messaging --version 1.36.0
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.0 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.0
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring ovirt-engine-sdk-4.2.3 because its extensions are not built. Try: gem pristine ovirt-engine-sdk --version 4.2.3
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring puma-4.3.3 because its extensions are not built. Try: gem pristine puma --version 4.3.3
Ignoring qpid_messaging-1.36.0 because its extensions are not built. Try: gem pristine qpid_messaging --version 1.36.0
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.0 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.0
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
cannot load such file -- semantic_puppet

[ERROR 2020-11-09T16:46:45 main] Could not get default values, cannot continue
       

When i run another upgrade check it shows me that the following services cannot be started dynflow-sidekiq@orchestrator, httpd, foreman-proxy

In journalctl:

Nov 09 17:03:28 evalsv0106 systemd[1]: dynflow-sidekiq: main process exited, code=exited, status=1/FAILURE
Nov 09 17:03:28 evalsv0106 systemd[1]: Unit dynflow-sidekiq entered failed state.
Nov 09 17:03:28 evalsv0106 systemd[1]: dynflow-sidekiq failed.
Nov 09 17:03:28 evalsv0106 dynflow-sidekiq@worker[2307]: 2020-11-09T16:03:28.405Z 2307 TID-hioxz ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
Nov 09 17:03:28 evalsv0106 dynflow-sidekiq@worker[2307]: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
Nov 09 17:03:29 evalsv0106 dynflow-sidekiq@worker[2307]: 2020-11-09T16:03:29.410Z 2307 TID-hioxz ERROR: Heartbeat thread error: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
Nov 09 17:03:29 evalsv0106 systemd[1]: dynflow-sidekiq holdoff time over, scheduling restart.
Nov 09 17:03:29 evalsv0106 systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
-- Subject: Unit dynflow-sidekiq has finished shutting down
-- Defined-By: systemd



Version-Release number of selected component (if applicable):

#rpm -qa satellite
satellite-6.8.0-1.el7sat.noarch

How reproducible: always


Steps to Reproduce:
rerun the upgrade


Is there any relation with this? https://community.theforeman.org/t/memory-recycler-in-the-age-of-sidekiq/20395

I tried this konfiguration but it did not help. What should I do? Thanks in advance
Comment 1 Steffen Froemer 2020-11-17 19:20:03 UTC 
Hi, I was able to reproduce this behavior. It seems to be related if puppet-enterprise was installed before.

If symlink '/usr/local/bin/puppet -> /opt/puppetlabs/bin/puppet' exist, the upgrade will fail with issue from above.


      ' | RUBYLIB=/opt/theforeman/tfm/root/usr/share/gems/gems/kafo-4.1.0/lib/kafo/../..//modules: /usr/local/bin/puppet apply --config=/tmp/kafo_installation20201117-31894-1pcop5h/puppet.conf 
[ERROR 2020-11-17T17:27:34 main] Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring ovirt-engine-sdk-4.2.3 because its extensions are not built. Try: gem pristine ovirt-engine-sdk --version 4.2.3
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring puma-4.3.3 because its extensions are not built. Try: gem pristine puma --version 4.3.3
Ignoring qpid_messaging-1.36.0 because its extensions are not built. Try: gem pristine qpid_messaging --version 1.36.0
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.0 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.0
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
Ignoring bcrypt-3.1.12 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.12
Ignoring ffi-1.12.2 because its extensions are not built. Try: gem pristine ffi --version 1.12.2
Ignoring nio4r-2.5.2 because its extensions are not built. Try: gem pristine nio4r --version 2.5.2
Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
Ignoring ovirt-engine-sdk-4.2.3 because its extensions are not built. Try: gem pristine ovirt-engine-sdk --version 4.2.3
Ignoring passenger-4.0.18 because its extensions are not built. Try: gem pristine passenger --version 4.0.18
Ignoring pg-1.1.4 because its extensions are not built. Try: gem pristine pg --version 1.1.4
Ignoring puma-4.3.3 because its extensions are not built. Try: gem pristine puma --version 4.3.3
Ignoring qpid_messaging-1.36.0 because its extensions are not built. Try: gem pristine qpid_messaging --version 1.36.0
Ignoring rkerberos-0.1.5 because its extensions are not built. Try: gem pristine rkerberos --version 0.1.5
Ignoring ruby-libvirt-0.7.0 because its extensions are not built. Try: gem pristine ruby-libvirt --version 0.7.0
Ignoring sqlite3-1.3.13 because its extensions are not built. Try: gem pristine sqlite3 --version 1.3.13
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
Ignoring unicode-0.4.4.4 because its extensions are not built. Try: gem pristine unicode --version 0.4.4.4
Ignoring websocket-driver-0.7.1 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.1
cannot load such file -- semantic_puppet

[ERROR 2020-11-17T17:27:34 main] Could not get default values, cannot continue


==> removing the link, upgrade was successful again.


Please find below my whole reproducer.

[root@satellite-1 ~]# ln -s /opt/puppetlabs/bin/puppet /usr/local/bin/puppet
[root@satellite-1 ~]# ls -la /usr/local/bin/puppet
lrwxrwxrwx. 1 root root 26 17. Nov 17:08 /usr/local/bin/puppet -> /opt/puppetlabs/bin/puppet




[root@satellite-1 ~]# satellite-maintain upgrade run --target-version 6.8
Checking for new version of satellite-maintain...
Nothing to update, can't find new version of satellite-maintain.
Running preparation steps required to run the next scenarios
================================================================================
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: 
\ Checking repositories enabled on the system                         [OK]      
--------------------------------------------------------------------------------


Running Checks before upgrading to Satellite 6.8
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check to make sure root(/) partition has enough space:                [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:                                 [OK]
--------------------------------------------------------------------------------
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for old tasks in paused/stopped state:                          [OK]
--------------------------------------------------------------------------------
Check for pending tasks which are safe to delete:                     [OK]
--------------------------------------------------------------------------------
Check for tasks in planning state:                                    [OK]
--------------------------------------------------------------------------------
Check to verify if any hotfix installed on system: 
| Checking for presence of hotfix(es). It may take some time to verify.         
                                                                      [OK]
--------------------------------------------------------------------------------
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: 
/ Checking repositories enabled on the system                         [OK]      
--------------------------------------------------------------------------------
Check if TMOUT environment variable is set:                           [OK]
--------------------------------------------------------------------------------
Check if any upstream repositories are enabled on system: 
\ Checking for presence of upstream repositories                      [OK]      
--------------------------------------------------------------------------------
Check if yum exclude list is configured:                              [OK]
--------------------------------------------------------------------------------
Check for roles that have filters with multiple resources attached:   [OK]
--------------------------------------------------------------------------------
Check for duplicate permissions from database:                        [OK]
--------------------------------------------------------------------------------
Check whether reports have correct associations:                      [OK]
--------------------------------------------------------------------------------
Check if checkpoint_segments configuration exists on the system:      [OK]
--------------------------------------------------------------------------------
Validate availability of repositories: 
/ Validating availability of repositories for 6.8                     [OK]      
--------------------------------------------------------------------------------


The pre-upgrade checks indicate that the system is ready for upgrade.
It's recommended to perform a backup at this stage.
Confirm to continue with the modification part of the upgrade, [y(yes), n(no), q(quit)] yes
Running Procedures before migrating to Satellite 6.8                            
================================================================================
disable active sync plans: 
\ Total 0 sync plans are now disabled.                                [OK]      
--------------------------------------------------------------------------------
Add maintenance_mode chain to iptables:                               [OK]
--------------------------------------------------------------------------------
Stop applicable services: 

Stopping the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy
\ All services stopped                                                [OK]      
--------------------------------------------------------------------------------


Running preparation steps required to run the next scenarios
================================================================================
Check if tooling for package locking is installed:                    [OK]
--------------------------------------------------------------------------------


Running Migration scripts to Satellite 6.8
================================================================================
Setup repositories: 
/ Configuring repositories for 6.8                                    [OK]      
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade: Could not get default values, check log file at /var/log/foreman-installer/satellite.log for more information
                                      [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 25
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.8] failed.

The following steps ended up in failing state:

  [installer-upgrade]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="installer-upgrade"


[root@satellite-1 ~]# unlink /usr/local/bin/puppet 
[root@satellite-1 ~]# satellite-maintain upgrade run --target-version 6.8
Checking for new version of satellite-maintain...
Security: kernel-3.10.0-1160.6.1.el7.x86_64 is an installed security update
Security: kernel-3.10.0-1062.el7.x86_64 is the currently running version
Nothing to update, can't find new version of satellite-maintain.
Running Checks before upgrading to Satellite 6.8
================================================================================
Skipping pre_upgrade_checks phase as it was already run before.
To enforce to run the phase, use `upgrade run --phase pre_upgrade_checks`



Running Procedures before migrating to Satellite 6.8
================================================================================
Skipping pre_migrations phase as it was already run before.
To enforce to run the phase, use `upgrade run --phase pre_migrations`



Running preparation steps required to run the next scenarios
================================================================================
Check if tooling for package locking is installed:                    [OK]
--------------------------------------------------------------------------------


Running Migration scripts to Satellite 6.8
================================================================================
Setup repositories:                                                   [ALREADY RUN]
The step was skipped as it was already run and it is marked as run_once. Use --force to enforce the execution.
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade: could not change directory to "/root"
Executing: scl enable rh-postgresql12 "PGSETUP_INITDB_OPTIONS='--lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8 --locale=en_US.UTF-8' postgresql-setup --upgrade"
 * upgrading from 'postgresql.service' to 'rh-postgresql12-postgresql.service'
 * Upgrading database.
 * Upgraded OK.
WARNING: The configuration files were replaced by default configuration.
WARNING: The previous configuration and data are stored in folder
WARNING: /var/lib/pgsql/data.
WARNING: If you've just upgraded your database from a previous major version of
Fedora or RHEL, please run reindexdb against your databases.  Core library
collation data may have changed and this will invalidate database indexes.  For
example, in Fedora 28 and RHEL 8 there have been extensive changes in glibc
collations to support ISO 14651:2016 (Unicode 9.0.0 data) and your indexes may
be affected: https://sourceware.org/ml/libc-announce/2018/msg00002.html
 * See /var/lib/pgsql/upgrade_rh-postgresql12-postgresql.log for details.
scl enable rh-postgresql12 "PGSETUP_INITDB_OPTIONS='--lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8 --locale=en_US.UTF-8' postgresql-setup --upgrade" finished successfully!
Executing: rm -f /etc/systemd/system/postgresql.service
rm -f /etc/systemd/system/postgresql.service finished successfully!
Preparing installation Done                                              
  Success!
  * Satellite is running at https://satellite-1.crazy.lab

  * To install an additional Capsule on separate machine continue by running:

      capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar"

  * To upgrade an existing 6.7 Capsule to 6.8:
      Please see official documentation for steps and parameters to use when upgrading a 6.7 Capsule to 6.8.

  * Capsule is running at https://satellite-1.crazy.lab:9090
  The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.
                                      [OK]
--------------------------------------------------------------------------------
Execute upgrade:run rake task:                                        [OK]
--------------------------------------------------------------------------------


Running Procedures after migrating to Satellite 6.8
================================================================================
Start applicable services: 

Starting the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
| All services started                                                [OK]      
--------------------------------------------------------------------------------
re-enable sync plans: 
/ Total 0 sync plans are now enabled.                                 [OK]      
--------------------------------------------------------------------------------
Remove maintenance_mode chain from iptables:                          [OK]
--------------------------------------------------------------------------------


Running Checks after upgrading to Satellite 6.8
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check if only installed assets are present on the system: 
\ Checking for presence of non-original assets...                     [OK]      
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------
Upgrade finished.
Comment 2 Ewoud Kohl van Wijngaarden 2020-11-18 14:46:11 UTC 
(In reply to Steffen Froemer from comment #1)
> If symlink '/usr/local/bin/puppet -> /opt/puppetlabs/bin/puppet' exist, the
> upgrade will fail with issue from above.

This is very likely the problem. In 6.8 the installer is running using the Ruby SCL (previously it was the system Ruby). That means there is code to de-SCL calls to Puppet. To allow testing with Puppet from gems, this is only done if the path to Puppet is detected as somewhere in /opt/puppetlabs. See https://github.com/theforeman/kafo/commit/a11bd9a45b67cdef61040693ea3bf614dc7265c9#diff-2b5409f20a69ddda045c05f4cdb5d8143a1f502b451a0f334ba716f429e6be88R47 for details.

We tried to skip symlinks, but files in /opt/puppetlabs/bin are also symlinks so that doesn't work. I'm not sure why we never tried to resolve the symlinks, that would have skipped it.

My current suggestion would be to remove the symlink in /usr/local/bin and run the installer. If that succeeds, we can confirm it's due to the issue I described.
Comment 3 Steffen Froemer 2020-11-18 15:05:57 UTC 
(In reply to Ewoud Kohl van Wijngaarden from comment #2)
> 
> My current suggestion would be to remove the symlink in /usr/local/bin and
> run the installer. If that succeeds, we can confirm it's due to the issue I
> described.

Just check my comment#1 

[root@satellite-1 ~]# unlink /usr/local/bin/puppet 
[root@satellite-1 ~]# satellite-maintain upgrade run --target-version 6.8
Checking for new version of satellite-maintain...
[... SKIPPING ...]
Running Checks after upgrading to Satellite 6.8
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check if only installed assets are present on the system: 
\ Checking for presence of non-original assets...                     [OK]      
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------
Upgrade finished.



==> That said, removing the link helps to succeed the upgrade/installation.
Comment 4 Ewoud Kohl van Wijngaarden 2020-11-18 15:57:56 UTC 
I did indeed see your comment just after I submitted. I think https://github.com/theforeman/kafo/pull/312 should help and allow the symlink to exist again. I'll see about spinning up a test environment to test it out.
Comment 8 Devendra Singh 2021-01-05 15:09:35 UTC 
Verified on 6.9 Snap 7.
Comment 12 errata-xmlrpc 2021-04-21 13:18:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313