Description of problem: Satellite upgrade from 6.10.2 (and 6.10.1) to 7.0 fails during db:seed with ActiveRecord::RecordNotDestroyed: Failed to destroy the record Empty satellite was used. Version-Release number of selected component (if applicable): Satellite 7.0.0 Snap6 How reproducible: 100% Steps to Reproduce: 1. Have a empty and registered Satellite 6.10.2 2. Upgrade to 7.0.0 # satellite-maintain upgrade run --target-version 7.0 -w repositories-validate,repositories-setup -y ... Running Migration scripts to Satellite 7.0 ================================================================================ Setup repositories: [SKIPPED] -------------------------------------------------------------------------------- Unlock packages: [OK] -------------------------------------------------------------------------------- Update package(s) : [OK] -------------------------------------------------------------------------------- Procedures::Installer::Upgrade: [FAIL] Failed executing LANG=en_US.utf-8 satellite-installer --disable-system-checks, exit status 6: ... 2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0] 2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0] ... Scenario [Migration scripts to Satellite 7.0] failed. The following steps ended up in failing state: [installer-upgrade] Actual results: failed upgrade due to failing db:seed Expected results: successful upgrade
2022-01-20 17:17:54 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:seed' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 'ErbParser' is ignored. 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 'RubyParser' is ignored. 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: API controllers newer than Apipie cache! Run apipie:cach e rake task to regenerate cache. 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted! 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordNotDestroyed: Failed to destroy the record 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/persistence.rb:951:in `_raise_record_not_destroyed' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/persistence.rb:551:in `destroy!' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/has_many_association.rb:99:in `each' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/has_many_association.rb:99:in `delete_records' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/collection_association.rb:385:in `remove_records' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/collection_association.rb:376:in `block in delete_or_destroy' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/collection_association.rb:135:in `block in transaction' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/collection_association.rb:134:in `transaction' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/collection_association.rb:376:in `delete_or_destroy' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor d-6.0.3.7/lib/active_record/associations/collection_association.rb:198:in `destroy' ... 2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0] 2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0] 2022-01-20 17:18:28 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Evaluated in 34.01 seconds
the failure seems to come from Katello: 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordNotDestroyed: Failed to destroy the record … 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/models/job_template.rb:68:in `import_raw!' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:10:in `block (3 levels) in <top (required)>' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:6:in `each' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:6:in `block (2 levels) in <top (required)>' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing' 2022-01-20 17:18:28 [INFO ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:3:in `block in <top (required)>'
Assigning to Repositories component based upon comment 3; however, feel free to update once specific cause is known.
Created redmine issue https://projects.theforeman.org/issues/34299 from this bug
Upstream bug assigned to jsherril
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34299 has been resolved.
VERIFIED. @Satellite 7.0.0 Snap8 foreman-3.1.1-1.el7sat.noarch by manual reproducer described in comment#0: # satellite-maintain upgrade run --target-version 7.0 -w repositories-validate,repositories-setup -y Checking for new version of satellite-maintain... Nothing to update, can't find new version of satellite-maintain. Running preparation steps required to run the next scenarios ================================================================================ Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: / Checking repositories enabled on the system [OK] -------------------------------------------------------------------------------- Running Checks before upgrading to Satellite 7.0 ================================================================================ Clean old Kernel and initramfs files from tftp-boot: [OK] -------------------------------------------------------------------------------- Check number of fact names in database: [OK] -------------------------------------------------------------------------------- Check whether all services are running: [OK] -------------------------------------------------------------------------------- Check whether all services are running using the ping call: [OK] -------------------------------------------------------------------------------- Check for paused tasks: [OK] -------------------------------------------------------------------------------- Check to verify no empty CA cert requests exist: [OK] -------------------------------------------------------------------------------- Check whether system is self-registered or not: [OK] -------------------------------------------------------------------------------- Check to make sure root(/) partition has enough space: [OK] -------------------------------------------------------------------------------- Check to make sure /var/lib/candlepin has enough space: [OK] -------------------------------------------------------------------------------- Check to validate candlepin database: [OK] -------------------------------------------------------------------------------- Check for running tasks: [OK] -------------------------------------------------------------------------------- Check for old tasks in paused/stopped state: [OK] -------------------------------------------------------------------------------- Check for pending tasks which are safe to delete: [OK] -------------------------------------------------------------------------------- Check for tasks in planning state: [OK] -------------------------------------------------------------------------------- Check to verify if any hotfix installed on system: \ Checking for presence of hotfix(es). It may take some time to verify. [OK] -------------------------------------------------------------------------------- Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: / Checking repositories enabled on the system [OK] -------------------------------------------------------------------------------- Check if TMOUT environment variable is set: [OK] -------------------------------------------------------------------------------- Check if any upstream repositories are enabled on system: / Checking for presence of upstream repositories [OK] -------------------------------------------------------------------------------- Check for roles that have filters with multiple resources attached: [OK] -------------------------------------------------------------------------------- Check for duplicate permissions from database: [OK] -------------------------------------------------------------------------------- Check if system has any non Red Hat RPMs installed (e.g.: Fedora): [OK] -------------------------------------------------------------------------------- Check whether reports have correct associations: [OK] -------------------------------------------------------------------------------- Check to validate yum configuration before upgrade: [OK] -------------------------------------------------------------------------------- Check if checkpoint_segments configuration exists on the system: [OK] -------------------------------------------------------------------------------- Validate availability of repositories: [SKIPPED] -------------------------------------------------------------------------------- The pre-upgrade checks indicate that the system is ready for upgrade. It's recommended to perform a backup at this stage. Confirm to continue with the modification part of the upgrade (assuming yes) Running Procedures before migrating to Satellite 7.0 ================================================================================ disable active sync plans: / Total 0 sync plans are now disabled. [OK] -------------------------------------------------------------------------------- Add maintenance_mode chain to iptables: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy \ stopping pulpcore-content.socket Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units. / All services stopped [OK] -------------------------------------------------------------------------------- Running preparation steps required to run the next scenarios ================================================================================ Check if tooling for package locking is installed: [OK] -------------------------------------------------------------------------------- Running Migration scripts to Satellite 7.0 ================================================================================ Setup repositories: [SKIPPED] -------------------------------------------------------------------------------- Unlock packages: [OK] -------------------------------------------------------------------------------- Update package(s) : [OK] -------------------------------------------------------------------------------- Procedures::Installer::Upgrade: [OK] -------------------------------------------------------------------------------- Execute upgrade:run rake task: [OK] -------------------------------------------------------------------------------- Running Procedures after migrating to Satellite 7.0 ================================================================================ Refresh detected features: [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy / starting pulpcore-content Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units. \ All services started [OK] -------------------------------------------------------------------------------- re-enable sync plans: | Total 0 sync plans are now enabled. [OK] -------------------------------------------------------------------------------- Remove maintenance_mode chain from iptables: [OK] -------------------------------------------------------------------------------- Running Checks after upgrading to Satellite 7.0 ================================================================================ Clean old Kernel and initramfs files from tftp-boot: [OK] -------------------------------------------------------------------------------- Check number of fact names in database: [OK] -------------------------------------------------------------------------------- Check whether all services are running: [OK] -------------------------------------------------------------------------------- Check whether all services are running using the ping call: [OK] -------------------------------------------------------------------------------- Check for paused tasks: [OK] -------------------------------------------------------------------------------- Check to verify no empty CA cert requests exist: [OK] -------------------------------------------------------------------------------- Check whether system is self-registered or not: [OK] -------------------------------------------------------------------------------- Check if only installed assets are present on the system: - Checking for presence of non-original assets... [OK] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Upgrade finished.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5498