Bug 1526863 - Broken content view blocks the 6.2->6.3 upgrade and leaves the system in a broken state
Summary: Broken content view blocks the 6.2->6.3 upgrade and leaves the system in a br...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.3.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: 6.4.0
Assignee: Justin Sherrill
QA Contact: Stephen Wadeley
URL:
Whiteboard:
Depends On:
Blocks: 1496794
TreeView+ depends on / blocked
 
Reported: 2017-12-18 03:34 UTC by Alexey Masolov
Modified: 2021-12-10 15:30 UTC (History)
17 users (show)

Fixed In Version: tfm-rubygem-katello-3.7.0.28-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-16 19:13:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 24768 0 Normal Closed updating errata counts on a content host should not use validation 2021-02-16 19:37:56 UTC

Description Alexey Masolov 2017-12-18 03:34:09 UTC
Description of problem:

'foreman-maintain upgrade run --target-version 6.3' fails if there is an orphaned content view in the system. It leaves the system in a broken state and it's impossible to get any info from it using hammer/etc. (ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system.). 
'foreman-maintain upgrade check --target-version 6.3' doesn't detect any problems.

Version-Release number of selected component (if applicable):
Satellite 6.3 Beta

How reproducible:
100%

Actual results:
====================================================================
$ foreman-maintain upgrade check --target-version 6.3

  == 20170913183848 AddErrataCounts: migrating ==================================
-- add_column(:katello_content_facets, :installable_security_errata_count, :integer, {:null=>false, :default=>0})
   -> 0.0328s
-- add_column(:katello_content_facets, :installable_enhancement_errata_count, :integer, {:null=>false, :default=>0})
   -> 0.0520s
-- add_column(:katello_content_facets, :installable_bugfix_errata_count, :integer, {:null=>false, :default=>0})
   -> 0.0316s
-- add_column(:katello_content_facets, :applicable_rpm_count, :integer, {:null=>false, :default=>0})
   -> 0.0338s
-- add_column(:katello_content_facets, :upgradable_rpm_count, :integer, {:null=>false, :default=>0})
   -> 0.0203s
foreman-rake db:migrate failed! Check the output for error!
Upgrade step migrate_foreman failed. Check logs for more information.
                                                                      [FAIL]
Failed executing satellite-installer --upgrade, exit status 1
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.3] failed.

The following steps ended up in failing state:

  [installer-upgrade]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="installer-upgrade"


$ hammer ping
ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system.

$ cat /var/log/foreman-installer/satellite.log
...
[DEBUG 2017-12-18 13:23:14 main] == 20170223114114 LockSeededTemplates: migrating ==============================rake aborted!
[DEBUG 2017-12-18 13:23:14 main] StandardError: An error has occurred, this and all later migrations canceled:
[DEBUG 2017-12-18 13:23:14 main] 
[DEBUG 2017-12-18 13:23:14 main] Validation failed: Content view 'atomic-test' is not in environment 'Library'/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:79:in `raise_record_invalid'
[DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:43:in `save!'
[DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
[DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `block in save!'
...

====================================================================

Expected results:
foreman-maintain complains before installation.

Additional info:

Details about the content view:

====================================================================
# hammer content-view info --id=21
ID:                     21
Name:                   atomic-test
Label:                  atomic-test
Composite:              false
Description:
Content Host Count:     0
Organization:           ANZLAB
Yum Repositories:
 1) ID:    2
    Name:  Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64
    Label: Red_Hat_Satellite_Tools_6_2_for_RHEL_7_Server_RPMs_x86_64
 2) ID:    1
    Name:  Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
    Label: Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server
 3) ID:    19
    Name:  Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64
    Label: Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x86_64
Docker Repositories:

OSTree Repositories:
 1) ID:    143
    Name:  Red Hat Enterprise Linux Atomic Host Trees
    Label: Red_Hat_Enterprise_Linux_Atomic_Host_Trees
Puppet Modules:

Lifecycle Environments:

Versions:

Components:

Activation Keys:


foreman=# select * from katello_content_views where id=21;
 id |    name     |    label    | description | organization_id | default |         created_at         |        updated_at         | composite | next_version
----+-------------+-------------+-------------+-----------------+---------+----------------------------+---------------------------+-----------+--------------
 21 | atomic-test | atomic-test |             |               1 | f       | 2017-11-13 00:04:14.121848 | 2017-11-13 00:09:44.89984 | f         |            2
(1 row)

foreman=# select * from katello_content_view_environments;
 id |        name         |        label        | cp_id | content_view_id |         created_at         |         updated_at         | environment_id | content_view_version_id
----+---------------------+---------------------+-------+-----------------+----------------------------+----------------------------+----------------+-------------------------
  1 | Library             | Library             | 1     |               1 | 2016-08-25 05:30:22.826564 | 2016-08-25 05:30:22.826564 |              1 |                       1
  2 | Library/ANZLab_Core | Library/ANZLab_Core | 1-2   |               2 | 2016-08-29 03:44:45.904437 | 2016-11-12 05:03:22.677069 |              1 |                      23
 13 | Library/Ceph_OSD    | Library/Ceph_OSD    | 1-19  |              19 | 2017-08-01 05:49:27.372094 | 2017-08-16 12:55:46.286284 |              1 |                      45
 14 | Library/sandpit     | Library/sandpit     | 1-20  |              20 | 2017-08-18 03:53:43.464042 | 2017-10-12 04:58:34.611849 |              1 |                      47
(4 rows)
====================================================================

Comment 5 Mike McCune 2018-01-29 18:20:12 UTC
Is this not a bug in the installer, not foreman-maintain?

Comment 6 Alexey Masolov 2018-01-29 22:42:33 UTC
I think this one is related to the both components actually

Comment 7 Ivan Necas 2018-02-08 11:45:16 UTC
I don't think there is anything right now in foreman-maintain to update to address this issue: this looks like an issue in migration code of the Katello, moving the component to contnet-management, with keyword set to Upgrades

Comment 8 Swapnil Abnave 2018-02-09 13:19:57 UTC
I think the orphaned records should not exist in the first place.

To me, this seems to be an issue with content-views, as the error was raised while running migrations. While calling content_facet.update_applicability_counts from migration AddErrataCounts.

I am more reluctant to set the component to content-views and create a separate RFE in foreman-maintain to fix/remove orphaned content-views.

Comment 10 sthirugn@redhat.com 2018-04-03 13:11:53 UTC
Removing target milestone - as per the comments, this needs to be handled outside of the foreman-maintain

Comment 11 Alexey Masolov 2018-04-26 01:23:19 UTC
In my opinion foreman-maintain should be able to check for the orphaned records and not going further with upgrading.

Comment 12 the.benparry 2018-07-19 11:31:08 UTC
We have exactly the same error during our Satellite 6.2 -> 6.3 upgrade.  Looking through your output I am unable to determine how you able to spot which version of the view is causing the issue?  We had to roll back to 6.2.15 which was a real pain.  We know which view/environment are causing the issue but I don't know how to identify it in the database to make sure the problem is gone before we try again.

Comment 13 Alexey Masolov 2018-07-19 23:27:16 UTC
The best course of actions would be removing the content view/all old versions probably.

Comment 18 Justin Sherrill 2018-08-30 22:52:27 UTC
Connecting redmine issue https://projects.theforeman.org/issues/24768 from this bug

Comment 28 Bryan Kearney 2018-10-16 19:13:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927


Note You need to log in before you can comment on or make changes to this bug.