Description of problem: 'foreman-maintain upgrade run --target-version 6.3' fails if there is an orphaned content view in the system. It leaves the system in a broken state and it's impossible to get any info from it using hammer/etc. (ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system.). 'foreman-maintain upgrade check --target-version 6.3' doesn't detect any problems. Version-Release number of selected component (if applicable): Satellite 6.3 Beta How reproducible: 100% Actual results: ==================================================================== $ foreman-maintain upgrade check --target-version 6.3 == 20170913183848 AddErrataCounts: migrating ================================== -- add_column(:katello_content_facets, :installable_security_errata_count, :integer, {:null=>false, :default=>0}) -> 0.0328s -- add_column(:katello_content_facets, :installable_enhancement_errata_count, :integer, {:null=>false, :default=>0}) -> 0.0520s -- add_column(:katello_content_facets, :installable_bugfix_errata_count, :integer, {:null=>false, :default=>0}) -> 0.0316s -- add_column(:katello_content_facets, :applicable_rpm_count, :integer, {:null=>false, :default=>0}) -> 0.0338s -- add_column(:katello_content_facets, :upgradable_rpm_count, :integer, {:null=>false, :default=>0}) -> 0.0203s foreman-rake db:migrate failed! Check the output for error! Upgrade step migrate_foreman failed. Check logs for more information. [FAIL] Failed executing satellite-installer --upgrade, exit status 1 -------------------------------------------------------------------------------- Scenario [Migration scripts to Satellite 6.3] failed. The following steps ended up in failing state: [installer-upgrade] Resolve the failed steps and rerun the command. In case the failures are false positives, use --whitelist="installer-upgrade" $ hammer ping ERF64-6496 [Foreman::MaintenanceException]: There are migrations pending in the system. $ cat /var/log/foreman-installer/satellite.log ... [DEBUG 2017-12-18 13:23:14 main] == 20170223114114 LockSeededTemplates: migrating ==============================rake aborted! [DEBUG 2017-12-18 13:23:14 main] StandardError: An error has occurred, this and all later migrations canceled: [DEBUG 2017-12-18 13:23:14 main] [DEBUG 2017-12-18 13:23:14 main] Validation failed: Content view 'atomic-test' is not in environment 'Library'/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:79:in `raise_record_invalid' [DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:43:in `save!' [DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/attribute_methods/dirty.rb:29:in `save!' [DEBUG 2017-12-18 13:23:14 main] /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `block in save!' ... ==================================================================== Expected results: foreman-maintain complains before installation. Additional info: Details about the content view: ==================================================================== # hammer content-view info --id=21 ID: 21 Name: atomic-test Label: atomic-test Composite: false Description: Content Host Count: 0 Organization: ANZLAB Yum Repositories: 1) ID: 2 Name: Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64 Label: Red_Hat_Satellite_Tools_6_2_for_RHEL_7_Server_RPMs_x86_64 2) ID: 1 Name: Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server Label: Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server 3) ID: 19 Name: Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64 Label: Red_Hat_Enterprise_Linux_7_Server_-_Extras_RPMs_x86_64 Docker Repositories: OSTree Repositories: 1) ID: 143 Name: Red Hat Enterprise Linux Atomic Host Trees Label: Red_Hat_Enterprise_Linux_Atomic_Host_Trees Puppet Modules: Lifecycle Environments: Versions: Components: Activation Keys: foreman=# select * from katello_content_views where id=21; id | name | label | description | organization_id | default | created_at | updated_at | composite | next_version ----+-------------+-------------+-------------+-----------------+---------+----------------------------+---------------------------+-----------+-------------- 21 | atomic-test | atomic-test | | 1 | f | 2017-11-13 00:04:14.121848 | 2017-11-13 00:09:44.89984 | f | 2 (1 row) foreman=# select * from katello_content_view_environments; id | name | label | cp_id | content_view_id | created_at | updated_at | environment_id | content_view_version_id ----+---------------------+---------------------+-------+-----------------+----------------------------+----------------------------+----------------+------------------------- 1 | Library | Library | 1 | 1 | 2016-08-25 05:30:22.826564 | 2016-08-25 05:30:22.826564 | 1 | 1 2 | Library/ANZLab_Core | Library/ANZLab_Core | 1-2 | 2 | 2016-08-29 03:44:45.904437 | 2016-11-12 05:03:22.677069 | 1 | 23 13 | Library/Ceph_OSD | Library/Ceph_OSD | 1-19 | 19 | 2017-08-01 05:49:27.372094 | 2017-08-16 12:55:46.286284 | 1 | 45 14 | Library/sandpit | Library/sandpit | 1-20 | 20 | 2017-08-18 03:53:43.464042 | 2017-10-12 04:58:34.611849 | 1 | 47 (4 rows) ====================================================================
Is this not a bug in the installer, not foreman-maintain?
I think this one is related to the both components actually
I don't think there is anything right now in foreman-maintain to update to address this issue: this looks like an issue in migration code of the Katello, moving the component to contnet-management, with keyword set to Upgrades
I think the orphaned records should not exist in the first place. To me, this seems to be an issue with content-views, as the error was raised while running migrations. While calling content_facet.update_applicability_counts from migration AddErrataCounts. I am more reluctant to set the component to content-views and create a separate RFE in foreman-maintain to fix/remove orphaned content-views.
Removing target milestone - as per the comments, this needs to be handled outside of the foreman-maintain
In my opinion foreman-maintain should be able to check for the orphaned records and not going further with upgrading.
We have exactly the same error during our Satellite 6.2 -> 6.3 upgrade. Looking through your output I am unable to determine how you able to spot which version of the view is causing the issue? We had to roll back to 6.2.15 which was a real pain. We know which view/environment are causing the issue but I don't know how to identify it in the database to make sure the problem is gone before we try again.
The best course of actions would be removing the content view/all old versions probably.
Connecting redmine issue https://projects.theforeman.org/issues/24768 from this bug
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2927