Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1526863

Summary: Broken content view blocks the 6.2->6.3 upgrade and leaves the system in a broken state
Product: Red Hat Satellite Reporter: Alexey Masolov <amasolov>
Component: Content ManagementAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Stephen Wadeley <swadeley>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3.0CC: amasolov, apatel, bbuckingham, cdonnell, dmoessne, egolov, inecas, jsenkyri, jsherril, kgaikwad, mmccune, nkathole, pdudley, smane, sthirugn, swadeley, the.benparry
Target Milestone: 6.4.0Keywords: PrioBumpGSS, Triaged, Upgrades
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.7.0.28-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-16 19:13:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1496794    

Description Alexey Masolov 2017-12-18 03:34:09 UTC
Description of problem:

'foreman-maintain upgrade run --target-version 6.3' fails if there is an orphaned content view in the system. It leaves the system in a broken state and it's impossible to get any info from it using hammer/etc. (ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system.). 
'foreman-maintain upgrade check --target-version 6.3' doesn't detect any problems.

Version-Release number of selected component (if applicable):
Satellite 6.3 Beta

How reproducible:
100%

Actual results:
====================================================================
$ foreman-maintain upgrade check --target-version 6.3

  == 20170913183848 AddErrataCounts: migrating ==================================
-- add_column(:katello_content_facets, :installable_security_errata_count, :integer, {:null=>false, :default=>0})
   -> 0.0328s
-- add_column(:katello_content_facets, :installable_enhancement_errata_count, :integer, {:null=>false, :default=>0})
   -> 0.0520s
-- add_column(:katello_content_facets, :installable_bugfix_errata_count, :integer, {:null=>false, :default=>0})
   -> 0.0316s
-- add_column(:katello_content_facets, :applicable_rpm_count, :integer, {:null=>false, :default=>0})
   -> 0.0338s
-- add_column(:katello_content_facets, :upgradable_rpm_count, :integer, {:null=>false, :default=>0})
   -> 0.0203s
foreman-rake db:migrate failed! Check the output for error!
Upgrade step migrate_foreman failed. Check logs for more information.
                                                                      [FAIL]
Failed executing satellite-installer --upgrade, exit status 1
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.3] failed.

The following steps ended up in failing state:

  [installer-upgrade]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="installer-upgrade"


$ hammer ping
ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system.

$ cat /var/log/foreman-installer/satellite.log
...
[DEBUG 2017-12-18 13:23:14 main] == 20170223114114 LockSeededTemplates: migrating ==============================rake aborted!
[DEBUG 2017-12-18 13:23:14 main] StandardError: An error has occurred, this and all later migrations canceled:
[DEBUG 2017-12-18 13:23:14 main] 
[DEBUG 2017-12-18 13:23:14 main] Validation failed: Content view 'atomic-test' is not in environment 'Library'/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:79:in `raise_record_invalid'
[DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:43:in `save!'
[DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
[DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `block in save!'
...

====================================================================

Expected results:
foreman-maintain complains before installation.

Additional info:

Details about the content view:

====================================================================
# hammer content-view info --id=21
ID:                     21
Name:                   atomic-test
Label:                  atomic-test
Composite:              false
Description:
Content Host Count:     0
Organization:           ANZLAB
Yum Repositories:
 1) ID:    2
    Name:  Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64
    Label: Red_Hat_Satellite_Tools_6_2_for_RHEL_7_Server_RPMs_x86_64
 2) ID:    1
    Name:  Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
    Label: Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server
 3) ID:    19
    Name:  Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64
    Label: Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x86_64
Docker Repositories:

OSTree Repositories:
 1) ID:    143
    Name:  Red Hat Enterprise Linux Atomic Host Trees
    Label: Red_Hat_Enterprise_Linux_Atomic_Host_Trees
Puppet Modules:

Lifecycle Environments:

Versions:

Components:

Activation Keys:


foreman=# select * from katello_content_views where id=21;
 id |    name     |    label    | description | organization_id | default |         created_at         |        updated_at         | composite | next_version
----+-------------+-------------+-------------+-----------------+---------+----------------------------+---------------------------+-----------+--------------
 21 | atomic-test | atomic-test |             |               1 | f       | 2017-11-13 00:04:14.121848 | 2017-11-13 00:09:44.89984 | f         |            2
(1 row)

foreman=# select * from katello_content_view_environments;
 id |        name         |        label        | cp_id | content_view_id |         created_at         |         updated_at         | environment_id | content_view_version_id
----+---------------------+---------------------+-------+-----------------+----------------------------+----------------------------+----------------+-------------------------
  1 | Library             | Library             | 1     |               1 | 2016-08-25 05:30:22.826564 | 2016-08-25 05:30:22.826564 |              1 |                       1
  2 | Library/ANZLab_Core | Library/ANZLab_Core | 1-2   |               2 | 2016-08-29 03:44:45.904437 | 2016-11-12 05:03:22.677069 |              1 |                      23
 13 | Library/Ceph_OSD    | Library/Ceph_OSD    | 1-19  |              19 | 2017-08-01 05:49:27.372094 | 2017-08-16 12:55:46.286284 |              1 |                      45
 14 | Library/sandpit     | Library/sandpit     | 1-20  |              20 | 2017-08-18 03:53:43.464042 | 2017-10-12 04:58:34.611849 |              1 |                      47
(4 rows)
====================================================================

Comment 5 Mike McCune 2018-01-29 18:20:12 UTC
Is this not a bug in the installer, not foreman-maintain?

Comment 6 Alexey Masolov 2018-01-29 22:42:33 UTC
I think this one is related to the both components actually

Comment 7 Ivan Necas 2018-02-08 11:45:16 UTC
I don't think there is anything right now in foreman-maintain to update to address this issue: this looks like an issue in migration code of the Katello, moving the component to contnet-management, with keyword set to Upgrades

Comment 8 Swapnil Abnave 2018-02-09 13:19:57 UTC
I think the orphaned records should not exist in the first place.

To me, this seems to be an issue with content-views, as the error was raised while running migrations. While calling content_facet.update_applicability_counts from migration AddErrataCounts.

I am more reluctant to set the component to content-views and create a separate RFE in foreman-maintain to fix/remove orphaned content-views.

Comment 10 sthirugn@redhat.com 2018-04-03 13:11:53 UTC
Removing target milestone - as per the comments, this needs to be handled outside of the foreman-maintain

Comment 11 Alexey Masolov 2018-04-26 01:23:19 UTC
In my opinion foreman-maintain should be able to check for the orphaned records and not going further with upgrading.

Comment 12 the.benparry 2018-07-19 11:31:08 UTC
We have exactly the same error during our Satellite 6.2 -> 6.3 upgrade.  Looking through your output I am unable to determine how you able to spot which version of the view is causing the issue?  We had to roll back to 6.2.15 which was a real pain.  We know which view/environment are causing the issue but I don't know how to identify it in the database to make sure the problem is gone before we try again.

Comment 13 Alexey Masolov 2018-07-19 23:27:16 UTC
The best course of actions would be removing the content view/all old versions probably.

Comment 18 Justin Sherrill 2018-08-30 22:52:27 UTC
Connecting redmine issue https://projects.theforeman.org/issues/24768 from this bug

Comment 28 Bryan Kearney 2018-10-16 19:13:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927