Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2019467

Summary: [RFE] Add an upgrade check for mismatches
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Satellite MaintainAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.10.0CC: ahumbe, apatel, ehelms, gscarbor, jyejare, kgaikwad, kkinge, mjia, nkim, peter.vreman, rlavi, saydas
Target Milestone: UnspecifiedKeywords: FutureFeature, Reopened, Upgrades
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-01 18:26:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2021-11-02 15:43:22 UTC
Description of problem:
User story: An upgrade from 6.9 to 6.10 beta failed during the upgrade process (due to taxonomy mismatch) on foreman-rake db:seed. That is something that is worth to pre-check before the upgrade, to prevent failed upgrades (esp. with such cryptic message).


Version-Release number of selected component (if applicable):
Sat6.10


How reproducible:
100%


Steps to Reproduce:
1. Have Sat6.9
2. Have one Org (Org1) with one Loc assigned to it, and another Org (Org2) _not_ assigned to the Loc.
3. Have a Host assigned to the Org1 and Loc1
4. Break the taxonomy by forcefully moving the host to the Org2, e.g. via rake commands:

host = Host.where(:name => 'testhost').first
host.organization_id      # to let print the ID
host.organization_id = 3  # to let update the ID
host.save!

(optionally, forcefully set it via foreman postgres db directly, via cmd like:

update hosts set organization_id = 3 where name = 'testhost';

)

5. Upgrade Sat to 6.10


Actual results:
5. Fails with backtrace:

2021-10-13 15:57:08 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:seed'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 'ErbParser' is ignored.
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 'RubyParser' is ignored.
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Locations expecting locations used by hosts or inherited (check mismatches report).
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:80:in `raise_validation_error'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/validations.rb:53:in `save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `block in save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:366:in `with_transaction_returning_status'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:318:in `save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:48:in `save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/app/lib/katello/lazy_accessor.rb:67:in `save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/models/foreman_tasks/concerns/action_triggering.rb:31:in `block in save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/models/foreman_tasks/concerns/action_triggering.rb:127:in `dynflow_task_wrap'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.3/app/models/foreman_tasks/concerns/action_triggering.rb:31:in `save!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/app/services/katello/organization_creator.rb:31:in `block in seed!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/transactions.rb:212:in `transaction'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/app/services/katello/organization_creator.rb:29:in `seed!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/app/services/katello/organization_creator.rb:8:in `block (2 levels) in seed_all_organizations!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/app/services/katello/organization_creator.rb:7:in `block in seed_all_organizations!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/app/services/katello/organization_creator.rb:6:in `seed_all_organizations!'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.15/db/seeds.d/102-organizations.rb:8:in `<top (required)>'
2021-10-13 15:57:39 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:318:in `load'
..


Expected results:
No such error _during_ upgrade, but the mismatch is identified in a pre-check


Additional info:

Comment 2 Sayan Das 2021-11-23 06:29:13 UTC
I actually came across the same issue on my own satellite, where it was due to domain.

2021-11-10 14:21:42 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
2021-11-10 14:21:42 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Doma
ins expecting domains used by hosts or inherited (check mismatches report). 


I had example.com as the domain. Two orgs created RedHat and SCA where, the domain was part of first org but was not with second.

Now a host was registered with SCA org i.e. host.example.com and if I am not incorrect, that host data was creating the problem in db:seed stage as the domain was not part of SCA org but the host was.


I could not figure out how to fix the mismatch as obviously, we don't have GUI working at that point. Only DB and rake console was working at that point and I could not found how to trigger the "Fix Mismatches" for an organization via rake. 

So, I went ahead and deleted that host from Satellite DB via rake console and then db:seed completed without any errors.

Obviously, we need to fix this within the installer but I like the idea of having a prerequisite mentioned in Sat 6.10 upgrade doc i.e. Check and click on "Fix Mismatches" for every single Organization and Location someone has in their satellite.

If everyone agrees, I can go for a DOC RFE here.

Comment 3 Sayan Das 2021-11-23 06:30:20 UTC
(In reply to Sayan Das from comment #2)
> I actually came across the same issue on my own satellite, where it was due
> to domain.
> 
> 2021-11-10 14:21:42 [INFO  ] [configure]
> /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:
> seed]/returns: rake aborted!
> 2021-11-10 14:21:42 [INFO  ] [configure]
> /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:
> seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Doma
> ins expecting domains used by hosts or inherited (check mismatches report). 
> 
> 
> I had example.com as the domain. Two orgs created RedHat and SCA where, the
> domain was part of first org but was not with second.
> 
> Now a host was registered with SCA org i.e. host.example.com and if I am not
> incorrect, that host data was creating the problem in db:seed stage as the
> domain was not part of SCA org but the host was.
> 
> 
> I could not figure out how to fix the mismatch as obviously, we don't have
> GUI working at that point. Only DB and rake console was working at that
> point and I could not found how to trigger the "Fix Mismatches" for an
> organization via rake. 
> 
> So, I went ahead and deleted that host from Satellite DB via rake console
> and then db:seed completed without any errors.
> 
> Obviously, we need to fix this within the installer but I like the idea of
> having a prerequisite mentioned in Sat 6.10 upgrade doc i.e. Check and click
> on "Fix Mismatches" for every single Organization and Location someone has
> in their satellite.
> 
> If everyone agrees, I can go for a DOC RFE here.

And yes, It could be great is some sort of rake method is available to trigger "Fix Mismatches" for Org or Locations from rake console. That could very much help in cases where upgrade is stuck in midway and we have no way to access UI to do the Fix Mismatches.

Comment 8 Sayan Das 2021-12-01 18:38:10 UTC
Please be aware of following BZ and PR as well.

2028205 – db:seed can fail when there are host mismatches
https://bugzilla.redhat.com/show_bug.cgi?id=2028205

Fixes #34059 - disable validation when seeding existing org by jturel · Pull Request #9822 · Katello/katello
https://github.com/Katello/katello/pull/9822/files

Comment 9 Jonathon Turel 2021-12-01 20:13:09 UTC

*** This bug has been marked as a duplicate of bug 2028205 ***

Comment 20 Brad Buckingham 2023-11-02 17:44:31 UTC
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team.  Thank you.

Comment 21 Brad Buckingham 2023-12-01 18:26:35 UTC
Thank you for your interest in Red Hat Satellite. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to contact your Red Hat Account Team. Thank you.