Bug 2106885

Description of problem:

Initial upgrade attempt to Satellite 6.11 fails to complete db:seed :

2022-07-11 14:36:24 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Database/obfuscateduser31::Rake[db:seed]/Exec[obfuscateduser31-rake-db:seed]: Starting to evaluate the resource (2269 of 2661)
2022-07-11 14:36:24 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Database/obfuscateduser31::Rake[db:seed]/Exec[obfuscateduser31-rake-db:seed]: '/usr/sbin/obfuscateduser31-rake db:seed' won't be executed because of failed check 'refreshonly'
2022-07-11 14:36:24 [DEBUG ] [configure] Exec[obfuscateduser31-rake-db:seed](provider=posix): Executing '/usr/sbin/obfuscateduser31-rake db:seed'
2022-07-11 14:36:24 [DEBUG ] [configure] Executing with uid=obfuscateduser31: '/usr/sbin/obfuscateduser31-rake db:seed'
2022-07-11 14:36:58 [INFO  ] [configure] /Stage[main]/obfuscateduser31::Database/obfuscateduser31::Rake[db:seed]/Exec[obfuscateduser31-rake-db:seed]/returns: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2022-07-11 14:36:58 [INFO  ] [configure] /Stage[main]/obfuscateduser31::Database/obfuscateduser31::Rake[db:seed]/Exec[obfuscateduser31-rake-db:seed]/returns: rake aborted!
2022-07-11 14:36:58 [INFO  ] [configure] /Stage[main]/obfuscateduser31::Database/obfuscateduser31::Rake[db:seed]/Exec[obfuscateduser31-rake-db:seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken

Now if upgrade will be re-executed, the db:seed execution will skip and installer fails on a different step:

2022-07-11 14:47:10 [DEBUG ] [configure] Foreman_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com](provider=rest_v3): Received response 422 from req
uest to https://bmp-sat-dev.obfuscatedkeyword0.com/api/v2/hosts?search=name%3D%22ah-1227211-001.sdi.corp.obfuscatedkeyword0.com%22
2022-07-11 14:47:10 [ERROR ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Could not evaluate: Error making GET request to obfuscateduser31 at https://bmp-sat-dev.obfuscatedkeyword0.com/api/v2/hosts: Response: 422 Unprocessable Entity
2022-07-11 14:47:10 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Evaluated in 0.55 seconds
2022-07-11 14:47:10 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_instance_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Starting to evaluate the resource (2553 of 2661)
2022-07-11 14:47:10 [INFO  ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_instance_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Dependency Foreman_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com] has failures: true
2022-07-11 14:47:10 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_instance_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Skipping because of failed dependencies
2022-07-11 14:47:10 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_instance_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Resource is being skipped, unscheduling all events
2022-07-11 14:47:10 [DEBUG ] [configure] /Stage[main]/obfuscateduser31::Register/Foreman_instance_host[obfuscateduser31-obfuscatedhost1.sdi.corp.obfuscatedkeyword0.com]: Evaluated in 0.00 seconds



Version-Release number of selected component (if applicable):

Satellite 6.11
Katello 4.3

How reproducible:

In certain customer environment


Steps to Reproduce:
1. Upgrade to 6.11
2.
3.

Actual results:

Same as mentioned in the case description.

A more clear output of db:seed will be as follows:

rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Name has already been taken
..
..
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation.rb:124:in `first_or_create!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.42/app/services/katello/organization_creator.rb:109:in `create_library_view'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.42/app/services/katello/organization_creator.rb:42:in `block in seed!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.42/app/services/katello/organization_creator.rb:35:in `seed!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.42/app/services/katello/organization_creator.rb:14:in `block (2 levels) in seed_all_organizations!'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation/delegation.rb:88:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/relation/delegation.rb:88:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.42/app/services/katello/organization_creator.rb:13:in `block in seed_all_organizations!'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.0.42/app/services/katello/organization_creator.rb:12:in `seed_all_organizations!'


Expected results:

No such issues.


Additional info:

Based on the traceback the issue is happening here:

https://github.com/Katello/katello/blob/beedffb2fef9868e434212e2c8815b4662b59f7e/app/services/katello/organization_creator.rb#L103-L110


    def create_library_view
      @library_view = Katello::ContentView.where(
        default: true,
        name: DEFAULT_CONTENT_VIEW_NAME,
        label: DEFAULT_CONTENT_VIEW_LABEL,
        organization: @organization
      ).first_or_create!
    end


https://github.com/Katello/katello/blob/beedffb2fef9868e434212e2c8815b4662b59f7e/app/services/katello/organization_creator.rb#L3-L4

    DEFAULT_CONTENT_VIEW_NAME = 'Default Organization View'.freeze
    DEFAULT_CONTENT_VIEW_LABEL = 'Default_Organization_View'.freeze

So if during db:seed installer founds the label is not matching with the name as set in code, it will try to recreate the view but will fail as an entry for name "Default Organization View" would already exist but with wrong label.

Upon checking the same, 

For the first organization only, I see this valid record with correct label

# echo "SELECT * FROM "katello_content_views" WHERE "katello_content_views"."label" = 'Default_Organization_View'"|su - postgres -c 'psql foreman'
 id |           name            |           label           | description | organization_id | default |         created_at         |         updated_at         | composite | next_version | force_puppet_environment | auto_publish | solve_dependencies | import_only | generated_for
----+---------------------------+---------------------------+-------------+-----------------+---------+----------------------------+----------------------------+-----------+--------------+--------------------------+--------------+--------------------+-------------+---------------
  1 | Default Organization View | Default_Organization_View |             |               1 | t       | 2019-09-13 11:52:36.429387 | 2019-09-13 11:52:36.429387 | f         |            1 | f                        | f            | f                  | f           |             0
(1 row)


But then for all the other organizations, I see entries like this with invalid label

# echo "SELECT * FROM "katello_content_views" WHERE "katello_content_views"."name" = 'Default Organization View'"|su - postgres -c 'psql foreman'

 id  |           name            |                label                 | description | organization_id | default |         created_at         |         updated_at         | composite | next_version | force_puppet_environment | auto_publish | solve_dependencies | import_only | generated_for
-----+---------------------------+--------------------------------------+-------------+-----------------+---------+----------------------------+----------------------------+-----------+--------------+--------------------------+--------------+--------------------+-------------+---------------
   2 | Default Organization View | e6688681-6312-4d96-8df6-1408824a305d |             |               3 | t       | 2019-09-16 13:08:04.779053 | 2019-09-16 13:08:04.779053 | f         |            1 | f                        | f            | f                  | f           |             0
 464 | Default Organization View | 9ed6e6ce-b902-4300-b54e-be8aa446f8b2 |             |             505 | t       | 2022-04-19 20:29:32.108608 | 2022-04-19 20:29:32.108608 | f         |            1 | f                        | f            | f                  | f           |             0

I have no idea how it can happen but following will fix the issue:

# echo "update katello_content_views set label = 'Default_Organization_View' where name = 'Default Organization View' and label <> 'Default_Organization_View';" | su - postgres -c 'psql foreman'

# foreman-maintain service restart

# satellite-installer --verbose