Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2043501 - Satellite upgrade fails during db:seed with ActiveRecord::RecordNotDestroyed: Failed to destroy the record
Summary: Satellite upgrade fails during db:seed with ActiveRecord::RecordNotDestroyed:...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Provisioning Templates
Version: 6.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.11.0
Assignee: Justin Sherrill
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-21 11:34 UTC by Lukas Pramuk
Modified: 2023-05-15 09:39 UTC (History)
6 users (show)

Fixed In Version: foreman-3.1.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-07-05 14:32:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 34299 0 Normal New upgrade fails during db:seed with ActiveRecord::RecordNotDestroyed: Failed to destroy the record 2022-01-21 16:36:08 UTC
Red Hat Product Errata RHSA-2022:5498 0 None None None 2022-07-05 14:32:32 UTC

Description Lukas Pramuk 2022-01-21 11:34:14 UTC 
Description of problem:
Satellite upgrade from 6.10.2 (and 6.10.1) to 7.0 fails during db:seed with ActiveRecord::RecordNotDestroyed: Failed to destroy the record 

Empty satellite was used.

Version-Release number of selected component (if applicable):
Satellite 7.0.0 Snap6

How reproducible:
100%

Steps to Reproduce:
1. Have a empty and registered Satellite 6.10.2  
2. Upgrade to 7.0.0

# satellite-maintain upgrade run --target-version 7.0 -w repositories-validate,repositories-setup -y
...

Running Migration scripts to Satellite 7.0
================================================================================
Setup repositories:                                                   [SKIPPED]
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [FAIL]
Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 6:
...

2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
...

Scenario [Migration scripts to Satellite 7.0] failed.

The following steps ended up in failing state:

  [installer-upgrade]


Actual results:
failed upgrade due to failing db:seed

Expected results:
successful upgrade
Comment 1 Lukas Pramuk 2022-01-21 11:43:36 UTC 
2022-01-20 17:17:54 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:seed'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 'ErbParser' is ignored.
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 'RubyParser' is ignored.
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: API controllers newer than Apipie cache! Run apipie:cach
e rake task to regenerate cache.
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordNotDestroyed: Failed to destroy the 
record
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/persistence.rb:951:in `_raise_record_not_destroyed'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/persistence.rb:551:in `destroy!'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/has_many_association.rb:99:in `each'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/has_many_association.rb:99:in `delete_records'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/collection_association.rb:385:in `remove_records'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/collection_association.rb:376:in `block in delete_or_destroy'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/collection_association.rb:135:in `block in transaction'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/collection_association.rb:134:in `transaction'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/collection_association.rb:376:in `delete_or_destroy'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecor
d-6.0.3.7/lib/active_record/associations/collection_association.rb:198:in `destroy'
...

2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-20 17:18:28 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' returned 1 instead of one of [0]
2022-01-20 17:18:28 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Evaluated in 34.01 seconds
Comment 3 Evgeni Golov 2022-01-21 12:19:42 UTC 
the failure seems to come from Katello:

2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordNotDestroyed: Failed to destroy the record
…
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-5.0.1/app/models/job_template.rb:68:in `import_raw!'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:10:in `block (3 levels) in <top (required)>'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:6:in `each'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:6:in `block (2 levels) in <top (required)>'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
2022-01-20 17:18:28 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0/db/seeds.d/75-job_templates.rb:3:in `block in <top (required)>'
Comment 4 Brad Buckingham 2022-01-21 13:17:22 UTC 
Assigning to Repositories component based upon comment 3; however, feel free to update once specific cause is known.
Comment 7 Justin Sherrill 2022-01-21 16:36:08 UTC 
Created redmine issue https://projects.theforeman.org/issues/34299 from this bug
Comment 8 Bryan Kearney 2022-01-21 20:05:22 UTC 
Upstream bug assigned to jsherril
Comment 9 Bryan Kearney 2022-01-21 20:05:24 UTC 
Upstream bug assigned to jsherril
Comment 10 Bryan Kearney 2022-01-24 08:05:12 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34299 has been resolved.
Comment 12 Lukas Pramuk 2022-02-09 11:07:33 UTC 
VERIFIED.

@Satellite 7.0.0 Snap8
foreman-3.1.1-1.el7sat.noarch

by manual reproducer described in comment#0:

# satellite-maintain upgrade run --target-version 7.0 -w repositories-validate,repositories-setup -y
Checking for new version of satellite-maintain...
Nothing to update, can't find new version of satellite-maintain.
Running preparation steps required to run the next scenarios
================================================================================
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: 
/ Checking repositories enabled on the system                         [OK]      
--------------------------------------------------------------------------------


Running Checks before upgrading to Satellite 7.0
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check to make sure root(/) partition has enough space:                [OK]
--------------------------------------------------------------------------------
Check to make sure /var/lib/candlepin has enough space:               [OK]
--------------------------------------------------------------------------------
Check to validate candlepin database:                                 [OK]
--------------------------------------------------------------------------------
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for old tasks in paused/stopped state:                          [OK]
--------------------------------------------------------------------------------
Check for pending tasks which are safe to delete:                     [OK]
--------------------------------------------------------------------------------
Check for tasks in planning state:                                    [OK]
--------------------------------------------------------------------------------
Check to verify if any hotfix installed on system: 
\ Checking for presence of hotfix(es). It may take some time to verify.         
                                                                      [OK]
--------------------------------------------------------------------------------
Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: 
/ Checking repositories enabled on the system                         [OK]      
--------------------------------------------------------------------------------
Check if TMOUT environment variable is set:                           [OK]
--------------------------------------------------------------------------------
Check if any upstream repositories are enabled on system: 
/ Checking for presence of upstream repositories                      [OK]      
--------------------------------------------------------------------------------
Check for roles that have filters with multiple resources attached:   [OK]
--------------------------------------------------------------------------------
Check for duplicate permissions from database:                        [OK]
--------------------------------------------------------------------------------
Check if system has any non Red Hat RPMs installed (e.g.: Fedora):    [OK]
--------------------------------------------------------------------------------
Check whether reports have correct associations:                      [OK]
--------------------------------------------------------------------------------
Check to validate yum configuration before upgrade:                   [OK]
--------------------------------------------------------------------------------
Check if checkpoint_segments configuration exists on the system:      [OK]
--------------------------------------------------------------------------------
Validate availability of repositories:                                [SKIPPED]
--------------------------------------------------------------------------------


The pre-upgrade checks indicate that the system is ready for upgrade.
It's recommended to perform a backup at this stage.
Confirm to continue with the modification part of the upgrade (assuming yes)
Running Procedures before migrating to Satellite 7.0                            
================================================================================
disable active sync plans: 
/ Total 0 sync plans are now disabled.                                [OK]      
--------------------------------------------------------------------------------
Add maintenance_mode chain to iptables:                               [OK]
--------------------------------------------------------------------------------
Stop applicable services: 

Stopping the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-api.socket, pulpcore-content.socket, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, foreman.socket, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
\ stopping pulpcore-content.socket                                              
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.
/ All services stopped                                                [OK]      
--------------------------------------------------------------------------------


Running preparation steps required to run the next scenarios
================================================================================
Check if tooling for package locking is installed:                    [OK]
--------------------------------------------------------------------------------


Running Migration scripts to Satellite 7.0
================================================================================
Setup repositories:                                                   [SKIPPED]
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [OK]
--------------------------------------------------------------------------------
Execute upgrade:run rake task:                                        [OK]
--------------------------------------------------------------------------------


Running Procedures after migrating to Satellite 7.0
================================================================================
Refresh detected features:                                            [OK]
--------------------------------------------------------------------------------
Start applicable services: 

Starting the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ starting pulpcore-content                                                     
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.
\ All services started                                                [OK]      
--------------------------------------------------------------------------------
re-enable sync plans: 
| Total 0 sync plans are now enabled.                                 [OK]      
--------------------------------------------------------------------------------
Remove maintenance_mode chain from iptables:                          [OK]
--------------------------------------------------------------------------------


Running Checks after upgrading to Satellite 7.0
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------
Check whether system is self-registered or not:                       [OK]
--------------------------------------------------------------------------------
Check if only installed assets are present on the system: 
- Checking for presence of non-original assets...                     [OK]      
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------
Upgrade finished.
Comment 17 errata-xmlrpc 2022-07-05 14:32:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498
Note You need to log in before you can comment on or make changes to this bug.